fastest way to read a text file in to a numpy array
michael.selik at gmail.com
Tue Jun 28 10:29:56 EDT 2016
On Tue, Jun 28, 2016 at 10:08 AM Hedieh Ebrahimi <hemla21 at gmail.com> wrote:
> File 1 has :
> and file2 has :
> I need to read the coordinates from file 1 and then interpolate a value
> for these coordinates on file 2 to the closest coordinate possible. The
> problem is file 2 is has around 5M lines. So I was wondering what would be
> the fastest approach?
Is this a one-time task, or something you'll need to repeat frequently?
How many points need to be interpolated?
How do you define distance? Euclidean 3d distance? K-nearest?
5 million can probably fit into memory, so it's not so bad.
NumPy is a good option for broadcasting the distance function across all 5
million labeled points for each unlabeled point. Given that file format,
NumPy can probably read from file directly into an array.
More information about the Python-list