# \[Resolved\] Inequality constraint for 2 datetime columns **URL:** \ **Category:** Synthetic Data Creation\ **Tags:** constraints, bug\ **Created:** [February 21, 2024, 7:44am UTC](https://forum.datacebo.com/t/resolved-inequality-constraint-for-2-datetime-columns/135 "2024-02-21T07:44:16Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![anna.popovychenko](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@anna.popovychenko](https://forum.datacebo.com/u/anna.popovychenko)\ **Post date:** [February 21, 2024, 7:44am UTC](https://forum.datacebo.com/t/resolved-inequality-constraint-for-2-datetime-columns/135/1 "2024-02-21T07:44:16Z")
Hello, I’m trying to apply below constraint for 2 datetime columns: ```auto subtr_close_date_ineq_constr = { 'constraint_class': 'Inequality', 'table_name': 'SubtrajectData', 'constraint_parameters': { 'low_column_name': 'openingsdatum', # opening_date 'high_column_name': 'afsluitdatum', # close_date 'strict_boundaries': False } } ``` Some of the values in “afsluitdatum” column are missing (NA). Example of those 2 columns: ```auto openingsdatum afsluitdatum 0 2022-08-30 2022-09-09 1 2020-07-28 2021-02-16 2 2022-03-05 2022-04-26 3 2020-06-16 2020-06-16 4 2021-06-05 2021-06-07 5 2020-07-26 2020-08-10 6 2023-01-24 7 2021-07-28 2021-08-09 8 2020-05-13 2020-07-13 9 2021-09-28 10 2020-01-27 2021-02-21 11 2022-12-19 2023-01-16 12 2019-07-24 2019-10-09 13 2020-06-27 14 2020-11-04 2021-01-15 15 2022-06-08 2022-06-21 16 2022-11-28 2022-11-28 17 2019-10-20 2022-02-14 18 2022-04-20 2022-05-05 19 2022-12-24 2023-01-23 ``` Metadata related part: ```auto "columns": { "openingsdatum": { "sdtype": "datetime", "datetime_format": "%Y-%m-%d" }, "afsluitdatum": { "sdtype": "datetime", "datetime_format": "%Y-%m-%d" }, ``` As result I get error: ```auto Step 2: Train synthesizer ... Traceback (most recent call last): File "C:\Users\anna.popovychenko\project\sdv_trial\fit.py", line 159, in synthesizer.fit(real_data) File "C:\Users\anna.popovychenko\.pyenv\pyenv-win\versions\3.9.13\lib\site-packages\sdv\multi_table\base.py", line 364, in fit processed_data = self.preprocess(data) File "packaging\\sdv_enterprise\\sdv\\multi_table\\hsa\\hsa.pyx", line 75, in sdv_enterprise.sdv.multi_table.hsa.hsa.expirable.wrapper File "packaging\\sdv_enterprise\\sdv\\multi_table\\hsa\\hsa.pyx", line 161, in sdv_enterprise.sdv.multi_table.hsa.hsa.HSASynthesizer.preprocess File "C:\Users\anna.popovychenko\.pyenv\pyenv-win\versions\3.9.13\lib\site-packages\sdv\multi_table\base.py", line 308, in preprocess self.validate(data) File "C:\Users\anna.popovychenko\.pyenv\pyenv-win\versions\3.9.13\lib\site-packages\sdv\multi_table\base.py", line 222, in validate raise InvalidDataError(errors) sdv.errors.InvalidDataError: The provided data does not match the metadata: boolean value of NA is ambiguous ``` But in docs it says that constraint ignores missing values ( [Inequality - Synthetic Data Vault (sdv.dev)](https://docs.sdv.dev/sdv/reference/constraint-logic/predefined-constraint-classes/inequality)) Could you help with that? Or it is expected behavior? Thanks ---
**Author:** ![neha](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.datacebo.com/neha/32/15_2.png) [@neha](https://forum.datacebo.com/u/neha)\ **Post date:** [February 21, 2024, 3:00pm UTC](https://forum.datacebo.com/t/resolved-inequality-constraint-for-2-datetime-columns/135/2 "2024-02-21T15:00:28Z")
Hi @anna.popovychenko, thanks for providing the details. The Inequality constraint is supposed to ignore missing values, so this is not working as expected. Unfortunately, I am unable to replicate the problem on my end. From the screenshots you shared, I suspect it has something to do with _how the data is represented in Python_. For some reason your missing values are showing as `` whereas usually they show up as `NaN`. # Short Term Workaround As a short-term workaround, I’d recommend converting the missing values: ```python import numpy as np real_data['SubtrajectData']['afsluitdatum'] = real_data['SubtrajectData']['afsluitdatum'].fillna(np.nan) ``` Let us know if that works. # Long Term To understand the root cause of this and fix the underlying bugs, it would be helpful to know _how_ you loaded in the data into Python. Was it from a CSV – or some other format? If you could share the code snippet of loading in `real_data`, that would be very helpful for replication. Thanks. ---
**Author:** ![anna.popovychenko](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@anna.popovychenko](https://forum.datacebo.com/u/anna.popovychenko)\ **Post date:** [February 21, 2024, 4:01pm UTC](https://forum.datacebo.com/t/resolved-inequality-constraint-for-2-datetime-columns/135/3 "2024-02-21T16:01:06Z")
Hi @neha , I tried `.fillna(np.nan)` - same result. For me helped below: ```auto real_data[['afsluitdatum', 'openingsdatum']] = \ real_data[['afsluitdatum', 'openingsdatum']].apply(pd.to_datetime) ``` I load data from csv files: ```auto real_data = pd.read_csv("file.csv") real_data= real_data.convert_dtypes() ``` In `real_data.dtypes` I get: ```auto openingsdatum string[python] afsluitdatum string[python] ``` ---
**Author:** ![neha](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.datacebo.com/neha/32/15_2.png) [@neha](https://forum.datacebo.com/u/neha)\ **Post date:** [February 21, 2024, 4:24pm UTC](https://forum.datacebo.com/t/resolved-inequality-constraint-for-2-datetime-columns/135/4 "2024-02-21T16:24:03Z")
Hi @anna.popovychenko, thanks for the info. I was able to replicate it now. The issue is with the `convert_dtypes` command. What is the intended reason for adding this command? If I delete the below line, then everything works fine. Could you try this? ```python # delete this line # real_data= real_data.convert_dtypes() ``` # Resources -Click this link for more information on how to load the data: **[Loading Data | Synthetic Data Vault](https://docs.sdv.dev/sdv/multi-table-data/data-preparation/loading-data)**. We have a built-in command called `load_csvs` that should load multiple files at once. No other conversion or modification should be needed. ```python from sdv.datasets.local import load_csvs # load multiple CSV files from within a folder data = load_csvs(folder_name='my_folder/') # the data is ready for training # no modifications should be needed unless we otherwise need a workaround synthesizer.fit(data) ``` ---
**Author:** ![anna.popovychenko](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@anna.popovychenko](https://forum.datacebo.com/u/anna.popovychenko)\ **Post date:** [February 22, 2024, 9:24am UTC](https://forum.datacebo.com/t/resolved-inequality-constraint-for-2-datetime-columns/135/5 "2024-02-22T09:24:07Z")
Hi @neha , thanks for recommendations. I’ve commented `convert_dtypes` line and removed all conversions for columns. For some of columns I get object type and if this column is a primary key then I get error during `fit()`. I added convertions only for such kind of columns. It works indeed. However I noticed increase in timing: for `fit()` from 260 to 614.9 seconds and for `sample()` from 53 to 300 seconds. It’s still very fast, just my observation. Why I used `convert_dtypes`: I was thinking that having columns with type `object` will slow down the process of `fit` and `sample`. As for `load_csvs` - thanks, it’s a good option ---
**Author:** ![neha](https://yyz2.discourse-cdn.com/flex030/user_avatar/forum.datacebo.com/neha/32/15_2.png) [@neha](https://forum.datacebo.com/u/neha)\ **Post date:** [April 29, 2024, 10:59pm UTC](https://forum.datacebo.com/t/resolved-inequality-constraint-for-2-datetime-columns/135/6 "2024-04-29T22:59:42Z")