Skip to content Skip to sidebar Skip to footer
Showing posts with the label Bigdata

Python Pandas Error While Removing Extra White Space

I am trying to clean a column in data frame of extra white space using command. The data frame has … Read more Python Pandas Error While Removing Extra White Space

Read From Line To Line Yelp Dataset By Python

I want to change this code to specifically read from line 1400001 to 1450000. What is modification?… Read more Read From Line To Line Yelp Dataset By Python

Why Is This Simple Spark Program Not Utlizing Multiple Cores?

So, I'm running this simple program on a 16 core multicore system. I run it by issuing the foll… Read more Why Is This Simple Spark Program Not Utlizing Multiple Cores?

Clean One Column From Long And Big Data Set

I am trying to clean only one column from the long and big data sets. The data has 18 columns, more… Read more Clean One Column From Long And Big Data Set

Get A List Of Subdirectories

I know I can do this: data = sc.textFile('/hadoop_foo/a') data.count() 240 data = sc.textFi… Read more Get A List Of Subdirectories

Sklearn-gmm On Large Datasets

I have a large data-set (I can't fit entire data on memory). I want to fit a GMM on this data s… Read more Sklearn-gmm On Large Datasets

Matrix Multiplication Using Hdf5

I'm trying to multiplicate 2 big matrices with memory limit using hdf5 (pytables) but function … Read more Matrix Multiplication Using Hdf5

Numpy: 3-byte, 6-byte Types (aka Uint24, Uint48)

NumPy seems to lack built-in support for 3-byte and 6-byte types, aka uint24 and uint48. I have a l… Read more Numpy: 3-byte, 6-byte Types (aka Uint24, Uint48)