Hi all, these changes came up in https://github.com/numpy/numpy/issues/21852 where a user had the use-case to look up the line number until where they want to read a file. The change is that `max_rows` now: * Represents the number of *rows* in the result * Gives a `UserWarning` when empty lines are skipped as a result. While previously `max_rows` used the number of *lines* (except those skipped initially). The difference is for a file formatted like: 1,2,3 # comment 2,3,4 The work-around to get the old version back is: import itertools lines = itertools.islice(open("file"), 0, max_rows) result = np.loadtxt(lines, ...) (Noted in the release notes and `UserWarning` – although the warning text could be improved.) There three possible "actions" I can think of: 1. We can add `max_lines` to do the `itertools` trick for the user. 2. The change is considered too big, we could revert it. 3. We could revert+deprecate the name for a new one, e.g. `nrows` and `nlines`. As an additional point of reference `pandas.read_csv` has `nrows` matching the new behavior. I do not have a strong opinion. I lean towards the new one, Chuck prefers the old meaning (I think). One reasoning for me was that users may also read too few data right now thinking `max_rows` has the new meaning already (i.e. we fix a bug for them). Cheers, Sebastian On Tue, 2022-02-08 at 08:08 -0600, Sebastian Berg wrote:
Hi all,
just a brief heads up that:
https://github.com/numpy/numpy/pull/20580
is now merged. This moves `np.loadtxt` to C. Mainly making it much faster. There are also some other improvements and changes though:
* It now supports `quotechar='"'` to support Excel dialect CSV. * Parsing some numbers is stricter (e.g. removed support for `_` or hex float parsing by default). * `max_rows` now actually counts rows and not lines. A warning is given if this makes a difference (blank lines). * Some exception will change, parsing failures now (almost) always give an informative `ValueError`. * `converters=callable` is now valid to provide a single converter for all columns.
Please test, and let us know if there is any issue or followup you would like to see.
We do have possible followups planned * Consider deprecating the `encoding="bytes"` default which exists for Python 2 compatibility. * Consider renaming `skip_rows` to the more precise `skip_lines`.
Moving to C unlocks possible further improvement, such as full `csv.Dialect` support. We do not have this on the roadmap, but such contributions are possible now. Similarly, it should be possible to rewrite `genfromtxt` based on this work.
Cheers,
Sebastian _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-leave@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: sebastian@sipsolutions.net