Suggested improvements to data downloads

Hi,

I'd like to get a bit of a discussion going about how downloading data from the site could be improved.

I am starting this thread, as I was looking at downloading Met Office weather data earlier this week. The form allows you to select a location and a set date, but this means you can only download data for one location (E.g. Cambridge) on one date. I'm interested in downloading all the data points within a county (multiple Met Office weather stations) over a date range.

It got me thinking about how the data could be interogated in other ways to make it more useful to the consumer.

I'm sure you all have examples of other datasets, you would like to interrogate in different ways; please share them here and hopefully the site administrators will be able to use our feedback to inform their roadmap.

Comments

Weather data

Hi,

This one is an interesting one, we setup the current service a while ago and we know it is far from perfect (in fact we are fixing some issues at the moment) we have budgeted for an overhaul of the service to provide much better access to the data and we would like to include you (or anyone who wishes to help) in testing the service as we develop it.  You can find the data through the Azure marketplace (the data is loaded from Azure) and there you have better options to play with it, although I believe an outlook.com/hotmail account is required.

I'll keep you posted on the project. In the meantime, if you wish to share a bit of the kind of functionality you would like in more detail, that would help us a lot.

Cheers,

 Flag as offensive 

Re: Weather Data

I'd be more than happy to help with testing.

As for greater detail around what I am after. Ideally I would like:

  • A way of identifying all locations that weather data is collected for by County - For example, looking at the current list of locations, Cambridge, Ely, Huntingdon, Ramsey etc. are all urban areas in the county of Cambridgeshire. Perhaps we could have an alternative pick list to select 'County', which would result in an export file containing all the geographically appropriate town/cities
  • If you wanted something slightly different to  a pick list, you could put in a basic GI interface which allows the user to select County polygons on a basic map. Maybe using one of the Ordnance Survey APIs.
  • Dates - At the moment you are limited to selecting one date. Ideally, I would like to export data between two dates
  • If possible, I would like to download a significant amount of data; I would like to be able to download 12 months data - 12 months as a calendar year / 12 months as a financial year / 12 months year to date
  • From memory, I think I read that the Met Office data goes back to 2011, Ideally i'd like to download more than 12 months
 Flag as offensive 

How about downloadable archives?

The data referred to is here:

http://data.gov.uk/metoffice-data-archive

which contains three datasets.

An 'observation data' download is only about 20KB, so a whole year's worth would be 170MB, which is manageable to download and analyse with a script or db - for Excel users you'd probably want to retain the existing form. Providing annual archives as simple downloads seems useful and dead easy to do.

The forecast data is substatially bigger. I'm not clear how its structured, so won't estimate the size, but each download is 42MB and 7MB. I imagine there are two clear use cases: 1. needing the most up-to-date forecast. Simply provide an unchanging URL where you can download the lastest info. 2. Researching historic forecasts - if there is a fresh load of data many times a day, these could stack up to GB per month, and this seems to be the only case where there are concerns over download size.

 Flag as offensive 

I would be interested in the

I would be interested in the 'Observation' data

 Flag as offensive 

me too, I'm after observation

me too, I'm after observation data 

 Flag as offensive 

Hi guys

Hi

Not sure if this is the best place to post but I added my question to an existing topic I found only to realise later it was over 5 years old so tried to create a new topic which hasnt gone live and i have had no reply to my chaser email to the address provided here so not sure how best to solicit feedback hence my post here to a more current generic discussion:-

I am trying to find out if there is currently any source of data for accessing phone, textphone and fax numbers for the many departmental offices shown throughout the Gov.UK website? Below is a tiny extract of current phone numbers available but there are hundreds more which potentially change/update over time making it difficult to maintain an accurate list. Does anything exist to help with this (XML file etc) to help ensure numbers are kept up to date easily and if not is anything planned? Any thoughts would be welcome. Thanks.

0300 790 6802 /  / 0300 200 3401 / 0345 300 3909 / 0300 200 3300 / 0300 200 3310 / 0844 318 0175 / 0345 302 1429 / 0844 272 4210 / 0845 712 3456 / 0844 318 0173 / 0300 200 3310 / 0844 272 4211 / 0300 790 6801 / 0844 318 0175 / 0300 200 3700 / 0844 318 0174 / 0300 790 6802 / 0844 272 4211 / 0300 200 3402 / 0300 200 3100 / 0844 272 4212 / 0345 300 3900 / 0345 300 3943 / 0844 272 4213 / 0844 272 4214 / 0300 200 3701 / 0844 272 4215 / 0345 300 3909

 Flag as offensive 

504 Errors

I'm usine Azure Marketplace as my data source for downloading data. So far I've been downloading historic Observation and DailyForecast data using urls. (I had tried to import data using Excel but the volumn of data makes it impractical - I tried waiting about 5 hours and gave up.)

Anyway my issue is that when I'm trying to download Forecast data I'm getting 504 errors, meaning the remote server threw 504 Gateway timeout. So it seems like the issue is on your side somehow. The errors happen when I apply a few filters to what I'm trying to download - adding a greater than date, rows to skip, and SiteIDs. Observation data is working fine with similar filters so it seems like it's maybe timing out because of the volumn of Forecast data it's trying to filter through. (I am obviously guessing here.)

Is it possible for someone from your side to look into this?

I noticed you mentioned testing earlier in the thread for some development you're doing to the downloading process. Let me know if I can help with that.

 Flag as offensive 

Improvement to data download

Hello,

This one is a stimulating one, we have a tendency to setup the present service a short while agone{and we have a tendency to|and that we have a tendency to} are aware of it is way from good (in reality we square measure fixing some problems at the moment) we've got budgeted for Associate in Nursing overhaul of the service to produce far better access to the information and that we would really like to incorporate you (or anyone UN agency needs to help) in testing the service as we develop it. you'll notice information|the info|the information} through the Azure marketplace (the data is loaded from Azure) and there you've got higher choices to play with it, though I think Associate in Nursing outlook.com/hotmail account is needed.

I'll keep you denote on the project. within the meanwhile, if you would like to share somewhat of the type of practicality you'd like in additional detail, that may facilitate United States plenty.

Keep it UP

 Flag as offensive 

Suggested improvements to data downloads

I tried to import the data using Excel but Volumn Contouring of data makes it impractical - I try to wait about 7 hours to give up

 Flag as offensive