[BASH] HTTP GET request

[FeralWombat]

Hello guys,

 

Full disclaimer, I have close to zero knowledge on anything in this topic.

I have an issue properly scripting a HTTP GET request. the API only allows for the 50 records at a time; to get results from the 51st record, we need to use a pagination variable "n", so n=51

I have a need to get about 2000 records, so roughly 40 queries, where the last one will be something like n=1951. A bash script can be created to increment the variable n until a limit and append the results of the query. However, this will mean that each start of a new page will begin with <?xml, which will break the xml file and make it unreadable in something like excel.

My question is: is there a way to continue with incrementing the variable n, but without adding the <?xml at the start of each page?

 

Thanks!

[Sauron]

I can think of multiple solutions for this:

 

• Save each page to a separate XML file
• Run the output through a regex to remove all <?xml tags except the first one (or just all of them and then add the first one manually)
• Start the concatenation of a new page from the 6th character
• Extract the data you need from each page and create your xml file with the desired format within the script

[FeralWombat]

That was a very quick reply! Sauron sees all?

Running the output through a regex sounds like the best option here. I don't know much about it but I'm sure old uncle G will help.

Thank you

[lal12]

Yeah for bash some kind of regular expression is probably the best solution. Otherwise use python or nodejs which can actually parse the result, so you can concat the different results afterwards.