Cleaner python3 example for use with gzip

[travc]

Not much of an 'issue', but...
The usage example with gzip could be better.

import ish_parser
import gzip

ish_filename = 'path/to/a/ish/file.gz'

# Read content
parser = ish_parser.ish_parser()
with gzip.open(ish_filename, 'rb') as gzstream:
    for line in gzstream:
        ishp.loads(line.decode('utf-8'))

# get the list of all reports
reports = parser.get_reports()
print(len(reports))

I'm currently working on code to fetch the data for a particular station over a particular range.
Also converting some results (just air_temperature for now) into pandas dataframes.
If your interested, I can submit that code to you if it ever gets done to a 'clean enough' standard.

Oh, and thanks for writing a parser for the insane mess... I was not looking forward to that.

[travc]

That fetching example (python3):

import ish_parser
import gzip
import ftplib
import io

# example from: ftp://ftp.ncdc.noaa.gov/pub/data/noaa/isd-history.txt
# 722950 23174 LOS ANGELES INTERNATIONAL AIR US   CA KLAX  +33.938 -118.389 +0029.6 19440101 20170509
USAF_ID = '722950'
WBAN_ID = '23174'
YEARS = range(1980, 1983) # so 1980, 1981, 1982

ftp_host = "ftp.ncdc.noaa.gov"
parser = ish_parser.ish_parser()

with ftplib.FTP(host=ftp_host) as ftpconn:
    ftpconn.login()

    for year in YEARS:
        ftp_file = "pub/data/noaa/{YEAR}/{USAF}-{WBAN}-{YEAR}.gz".format(USAF=USAF_ID, WBAN=WBAN_ID, YEAR=year)
        print(ftp_file)

        # read the whole file and save it to a BytesIO (stream)
        response = io.BytesIO()
        try:
            ftpconn.retrbinary('RETR '+ftp_file, response.write)
        except ftplib.error_perm as err:
            if str(err).startswith('550 '):
                print('ERROR:', err)
            else:
                raise

        # decompress and parse each line 
        response.seek(0) # jump back to the beginning of the stream
        with gzip.open(response, mode='rb') as gzstream:
            for line in gzstream:
                parser.loads(line.decode('latin-1'))
                
# get the list of all reports
reports = parser.get_reports()
print(len(reports), "records")     

pub/data/noaa/1980/722950-23174-1980.gz
pub/data/noaa/1981/722950-23174-1981.gz
pub/data/noaa/1982/722950-23174-1982.gz
29792 records
CPU times: user 3.83 s, sys: 152 ms, total: 3.98 s
Wall time: 13 s

And combining with pandas (in jupyter notebook)

# using reports variable generated by code above
import pandas as pd

foo = pd.DataFrame.from_records(((r.datetime, r.air_temperature.get_numeric()) for r in reports),
                         columns=['datetime','AT'],
                         index='datetime')
foo.index = pd.to_datetime(foo.index) # convert the index to pandas datetime objects
foo.plot()

[haydenth]

Super cool. I would love to have better integration with dumping the data out to a pandas dataframe. I don't have much experience specifically with pandas, so let me look into this today. I'm glad this is a helpful library

[travc]

Just an FYI: I updated the code in the previous comment...

Turns out urllib was triggering 'Max Number of Connections' errors... Apparently opening and closing the connection each time, without waiting/forcing the previous connections to close even when close() is explicitly called. ftplib is more appropriate to the task.

[haydenth]

Yeah, I download all the files in advance using ncftp first and then use this library to parse it, so I don't have to worry about that. Here's how I do it using recursive ftp + http to NOAA's servers: https://unix.stackexchange.com/questions/177279/speeding-up-recursive-ftp/178464#178464