AngularJS Tutorial:
Working With a Large Dataset

November 24, 2015

I was curios to how AngularJS would perform using a large client-side dataset. Another word, can you display, filter, and search a very large array of objects (e.g., 1,000,000 records)? What library would you use? How responsive will your app be? In this tutorial we'll find out.

The tutorial assumes you have a large JSON data file that you need to process using AngularJS (or plain JS). This may or may not be a realistic scenario and, ideally, in the data-driven app paradigm you should have your back-end data service (or database) handle data requests and provide only a small dataset that your front-end application will use (based on user interactions with the app).
However, in the off chance you do end-up with a large dataset that needs to be processed on the client-side, you may find this tutorial useful.

Complete source code available here: https://github.com/glebpopoff/angularjs-large-dataset-performance
The actual app is hosted on AWS S3: https://s3-us-west-2.amazonaws.com/angularjs-apps/large-dataset-demo-1M-records/index.html

Generating Data

I used Python to generate sample data. The JSON dataset was generated in the following format (below). I tested the app using 10,000, 100,000, and 1,000,000 records.

Python is an awesome language. With Python it's pretty easy to randomly generate strings and work with lists. Basically, the data generator script creates a Customer JSON object with auto generated data. I decided to hardcode State and Company data to make it easier to search and sort.

Fetching Data

I used the AngularJS $http service to retrieve data from the server. It did take some time to download a large data file from the server; I don't think there's a way to speed up the download process on the client-side (i.e., the browser still needs to download the actual JSON file to your browser).

The generated file sizes:
1,000,000 records - 144.4MB
100,000 records - 14.3MB
1,000 records - 1.4MB

AngularJS Service to retrieve the dataset from server.

Processing Data

To process the dataset I used the plain JS sorting/searching functions while also benchmarking my results against the lodash.js utility functions.

Lodash is pretty cool and you can do quite a bit with just a single line of code. For example, sorting an array of objects can be as simple as this:
Using plain JS you have to write a little more code to accomplish the same goal:

Because the size of our data can be quite large we can't really use an otherwise awesome AngularJS Template Filter library. When I tried adding NG Filter to my array of 100,000 records the browser almost crashed (Chrome) and the page became unresponsive.

To perform a basic search against the dataset I implemented a quick linear function while also using the Lodash.JS _filter function for benchmarking. The code below was tested with an array of up to 1,000,000 JSON objects. JavaScript is actually amazing good at handling large arrays.. And the lodash.js version:

Loading Large Data Files

You may need to tell the user that the app is trying to load a large data asset (e.g., using $http service) and display a message that there's an ongoing background data fetching task.

I typically use the following HTTP interceptor code (inside app.js) that then triggers a custom directive to display the "Loading..Please wait" message.
And the directive code looks as following:

Results

The application performed quite well against 10,000 and 100,000 records (under a second sorting/searching). With 1,000,000 records things got a little bit more interesting and I started seeing execution time anywhere between 4-7 seconds when sorting. Lodash.js seems to perform better on a large dataset.
The app can be tested using the following link: https://s3-us-west-2.amazonaws.com/angularjs-apps/large-dataset-demo-1M-records/index.html

Full source code can be pulled from this repo: https://github.com/glebpopoff/angularjs-large-dataset-performance.
Email me () if you have any questions or suggestions. Written by GlebP, 2015.