Assuming variable names in our code

[leifdenby]

The problem

To able to make a nice profile plot function with annotated heights (like #15) the necessary physical variables will need to be present in the dataset (ds), so that when the user provides:

def plot_profiles_1d(ds, variables=["T", "rh", "qv"], height_labels=["z_RHmax", "z_INV"]):

the functions that calculate z_RHmax and z_INV can find the correct fields inside ds.

I think there are basically two ways of doing this:
a) We assume what the fields should be called and require the user to
rename fields (or maybe set the standard_name) attribute
b) We pass in the mapping for the variables as we do it not with kwargs.

I'm worried that option b) might get quite messy. This is because we're here calling a plotting function and then that plotting function calls the underlying function required to calculate the different heights (for example z_maxRH). So we'd need to define a mapping that the plot function can pass down to z_maxRH. If we're making use of a mapping anyway maybe we should just make the user rename things on their dataset with e.g.

ds_fixed = ds.rename(dict(temp="temperature", RH="rh"))

Suggested solution

I think we should go with option a). What we could do to make this nicer for the user is:

1. Inform the user what fields are missing and what that field should contain physically, i.e. we'd check for the existence of a variable in a dataset and then raise an exception like:

raise Exception("Couldn't find the variable `{var_name}` in the provided dataset."
    "To use {plot_function_name} you need to provide the "
    "`{field_description}` as `{var_name}`")

2. Optionally, we could allow the user to use CF-convention "standard name" attributes for variables. That way they don't need to rename fields, but simply set the correct "standard name", which many data files already will have.

Implementation details

To make all this work I was thinking we could simply use "standard names" in our code (based on cf-conventions where they exist) and make a module that defines this nomenclature. Something like this:

eurec4a_environment/nomenclature.py:

RELATIVE_HUMIDITY = "relative_humidity"
TEMPERATURE = "air_temperature"

this could be used inside a function as:

import eurec4a_environment.nomenclature as nom

def calc_maxRH_height(ds, z_min=200, z_max=900):
   da_rh = _get_field(ds, v=nom.RELATIVE_HUMIDITY)

And _get_field would then handle the checking for fields, returning them and raising helpful exceptions.

Thoughts?

[leifdenby]

@annalea-albright, @Geet-George, @JuleRadtke it would be great to have your thoughts on this :D

[JuleRadtke]

We can talk about it in 5 minutes?

[Geet-George]

I completely agree. Most datasets would (should) be conforming to CF-convention, and it would be easier to use the standard_name across the datasets. There might be some variables that don't have a CF standard_name, but these are rare and we can worry about them later.

I am having the same issue with user-defined levels (and to parse the dataset for variables). The resolution of this issue will solve my problem too. If some one is not already working on this, I am happy to take it up.

[leosaffin]

I'm wondering if I can link in the code that I wrote for #16 to this. (I'm not sure, because it requires a rethink of the way the functions are currently written.) It worked well with the JOANNE dataset because I set it up to look for the CF standard_name.

e.g. with the find_LCL_Bolton, we could instead do something like

from eurec4a_environment.interface import calculate
lcl = calculate("atmosphere_lifting_condensation_level", dataset)

However, the function find_LCL_Bolton would have to be modified to be called as

find_LCL_Bolton(temperature, rh, altitude)

and you add an entry in eurec4a_environment.interface.available

atmosphere_lifting_condensation_level=dict(
function=find_LCL_Bolton,
arguments=["air_temperature", "relative_humidity", "altitude"],
)

I think this could be better because:

1. You remove any assumption of specific variable names from the function leaving the user to either
   a. decide which variables to use and directly input them to find_LCL_Bolton
   b. call the calculate function
2. You don't need to write in the code for extracting a variable from a dataset for every new function you write
3. I think this might complement the approach in Levels #21 . i.e. that function could make use of calling this one

Is this worth pursuing. There are still a bunch of limitations, which I can work through in #16 . The main one I can think of is that it currently doesn't allow for multiple definitions of diagnostics but I think that shouldn't be too difficult to work out.

[Geet-George]

I really like Leo's approach. It's quite flexible, and does not explicitly ask the user to rename variables, in case they already know what they want to work on. If this could be incorporated, I guess changing all functions to only use variable arguements should not be a difficult job. It would definitely help #21.

[leifdenby]

I agree that having the call signature be:

find_LCL_Bolton(temperature, rh, altitude)

Avoids the user having to name things in a particular way, which is nice, but:

1. I do quite like the concept of passing in the whole dataset, it's very compact and ensures that all variables exist in the same coordinate system (@annalea-albright used this in her original code, and Robert also suggested this as an idea). It mimics the behaviour of seaborn and pandas where you call by plot_func(data=df, x="temperature", y="height") for example

2. I think that with what you propose @LSaffin the correct naming would be required when using calculate("atmosphere_lifting_condensation_level", dataset) say, and so in extension will be required when making plots (annotated with heights or layers) too. I think this could lead to confusion, being able to in one case pass in variables in one way but then when plotting having to rename/add standard_name attributes.

[leosaffin]

2-I think the way calculate is written would still work here. It does three possible things
a. If you request a name that is already in the dataset it just returns it
b. If not, it checks whether the requested name matches a CF standard name in the dataset
c. If it can't do either of those it checks whether it can calculate the variable from other variables that can be got from the dataset (recursively so it can calculate other variables needed as input)
"a" ensures that you can reliably call this function with non-standard names if that is how they are in your dataset so should be able to be reliably used in a plot_function. "b" and "c" would have to be manually done otherwise anyway.

1-Even if you do this style with the full dataset being passed instead, I think you should use CF standard naming anyway otherwise you are just creating an extra standard that will only be consistent with JOANNE. With a standard naming and the plot_func(data=df, x="temperature", y="height") you have more chance that people will be able to call the function without having to specify the names. You could then just use steps "a" and "b" from calculate so it looks for the standard_name attribute only if it can't already find the variable in the dataset.

[leifdenby]

Sorry I'm being slow, but I still don't quite get it... If I called the plotting function

plot_profile_2D(ds, variable="rh", y="height", x="time", levels=["z_RHmax", "z_INV"])

the user would have to ensure that the variables inside of ds have the correct variable names (or standard_names) for the functions that produce z_RHmax and z_INV, right? The function for z_INV might need the temperature for example.

[leosaffin]

@leifdenby - Ah, I see how that is an issue but it's a separate issue to the calculate function. With the current call you described the z_RHmax and z_INV function are going to have to assume some kind of naming, calculate will just make the matching of that name more flexible.

If you wanted to fix that in a seaborn style then the function would also have the optional arguments z_RHmax_kws and z_INV_kws. Then you would pass in a dictionary for each of these with any names you need to specify. (I'm not a fan of seaborn for this though because this ends up buried deep when you try to navigate the documentation.)

[leosaffin]

Maybe since the levels are specified as a list of strings, you would instead pass a levels_kws which would be a dictionary with an entry for each argument you pass to levels. This is me thinking "seaborn style" not my actual suggestion

[annalea-albright]

Sorry for the radio silence, and thanks for initiating this discussion! I was struggling with how to name and call variables. For now, I just defined the dimension names following the Joanne and radiosonde convention, i.e. plot func(ds, altitude="height", time="launch_time" ...), but this won't hold for other platforms ... I defer to you guys for how best to structure this, as you have more experience!

[leosaffin]

@annalea-albright - I think it's good to start off with the JOANNE naming so you can get the functions down and working. Carry on with that because if a different format is decided it will be easy to do some automated search/replace given how well it is currently written.

[annalea-albright]

@LSaffin sounds like a plan :)

[leifdenby]

thanks for the discussion here @annalea-albright, @JuleRadtke, @LSaffin and @Geet-George. Based on what we've arrived at I'm going to implement as option 2) above. We enforce that a variable should either a) have a particular name (e.g. temperature, I will use the JOANNE variable names here) or b) have the correct standard name (e.g. air_temperature, as JOANNE does). I will add a function that fetches a field from a dataset and provides helpful error messages if it can't find the field.

One aspect of this is that multiple fields in a dataset might have the standard_name air_temperature. I'm going to return all the variables for now if the variables are in the same coordinates and then add a dimension that was the previous variable name. This will make it possible to pass in two measurements of temperature for example and get boundary-layer height etc for both (so extra definitions on the same variable will just be treated as another dimension in this way, like time).

Hope this all sounds ok. I'll make a pull-request 😄