Python-Based Machine Learning (ML) Hadoop System
Licensed software, source code (C#, Python) and configuration files, serving as a foundation for the further development of a Machine Learning (ML) Hadoop system.
License Inquiry
What the License Includes
- What the License IncludesA ready-to-use system for working with data triggers in Hadoop, with ML-based control.
- Source CodeC# and Python source code.
- Configuration FilesIncluding the “In-Memory OLTP” configuration.
Procedure for Working with a Set of Data Triggers in Hadoop
The procedure runs at a defined time period (DTP), for example, initially 1 second. The ML system can periodically adjust the DTP based on the nature of the data (e.g., the running average).
- Creating an external table from the Hadoop data on which the triggers are based. One external table contains 1 to n triggers, i.e., it represents a set of triggers. There may be groups of such external tables, created from different data sets. Each data set consists of Hadoop data with a single “date-timestamp” parameter, which serves as the Primary Key of the external table.
- Adding a new row to a table with the same structure, but in-memory. This creates a multi-row table with the “date-timestamp” parameter as its Primary Key.
- Comparing the most recently added row with the previous one. The result of the comparison is equal or different.
- On different , the assigned Stored Procedure is executed for the corresponding trigger (the name of the Stored Procedure to run when the content changes is specified when the trigger is defined). The difference between the “date-timestamp” values of the two rows (PM2P) is passed to the ML system.
ML System
The ML system consists of an interface to the trigger procedure (written in Python, described above), which connects through Python control code to a Python library such as Scikit-learn or TensorFlow:
- New PM2P (received from the procedure)
- Trained PM2P (received from the Python library)
- DTP = { mean value | (trained PM2P) / 4 | … }
- The new DTP is passed to the procedure
In-Memory OLTP Configuration
Configuration steps:
- Identifying the in-memory table
- Creating Columnstore indexes for each trigger
- Connecting SQL Server 2017 Analysis Services to the in-memory table
Interested in an ML Hadoop License?
Contact us to discuss the license terms and deployment in your environment.
Contact us