Release candidate

device-detection-specification/pipeline-elements/hardware-profile-lookup-cloud.md

@@ -23,6 +23,9 @@ Currently, this capability is used for two use-cases:
   [Android devices](https://developer.android.com/reference/android/os/Build#MODEL)
   and [iOS devices](https://gist.github.com/soapyigu/c99e1f45553070726f14c1bb0a54053b#file-machinename-swift)
 
+The same lookup without a remote server is described in
+[hardware profile lookup on-premise](hardware-profile-lookup-on-premise.md).
+
 ## Accepted Evidence
 
 None. This Engine uses the output from the Cloud Aspect Engine as input

device-detection-specification/pipeline-elements/hardware-profile-lookup-on-premise.md

@@ -0,0 +1,136 @@
+# Hardware profile lookup on-premise
+
+## Overview
+
+This On-premise Engine gives the results of the
+[hardware profile lookup](hardware-profile-lookup-cloud.md) without a remote
+server. Given a TAC or a native model name, it returns every hardware
+[profile](device-detection-on-premise.md) in the data file whose TAC or
+NativeModel Property holds that value.
+
+It uses the data file of the
+[Device Detection on-premise](device-detection-on-premise.md) Engine. It needs a
+data file that holds the TAC and NativeModel Properties. The published
+property list of the
+[Lite data file](https://github.com/51Degrees/data#device-detection) has
+neither.
+
+## Native component
+
+The lookup is part of the native component in
+[51degrees-cxx/src/dd/hash](https://github.com/51Degrees/51degrees-cxx/tree/main/src/dd/hash),
+added in 51degrees-cxx commit
+[b4d2f268](https://github.com/51Degrees/51degrees-cxx/commit/b4d2f268).
+The Device Detection on-premise Engine gives it as the
+[profiles for a property value](device-detection-on-premise.md#profiles-for-a-property-value),
+and this Engine is built on that.
+
+### Index of profiles by value
+
+When the native data set is initialized, it builds an index from each value of
+the Properties the caller names to the profiles holding that value. The index
+is part of the data set. No Property is indexed unless the caller names it,
+whatever the performance profile. A named Property is indexed whether or not
+it is a required Property, so a caller does not need to require a Property to
+look up its values. A name that the data file has no Property of indexes
+nothing.
+
+When the data is refreshed, the new data set indexes the same Properties from
+the new data file, and the index of the data set it replaces is freed with
+that data set.
+
+Without the index, or for a Property that was not named, the lookup reads
+every profile and returns the same profiles more slowly.
+
+### Results
+
+A lookup MUST return every profile holding the value, each once, in the order
+of the profiles in the data file, which is ascending profile id. A value that
+no profile holds returns no profiles. The Property name and the value are
+matched exactly, as the data file holds them, including their case, as the
+[profiles for a property value](device-detection-on-premise.md#profiles-for-a-property-value)
+require.
+
+Choosing one of several profiles is left to the caller. A caller that wants a
+single device MAY pass the id of one profile to the Device Detection
+on-premise Engine as the `51D_ProfileIds` Evidence.
+
+### What each language exposes
+
+| **Language**                      | **Properties to index**                                                                                                          | **Lookup**                                                                                        | **Returns**                                                                |
+|-----------------------------------|----------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------|
+| C                                 | `fiftyoneDegreesHashInitManagerFromFileWithIndexedProperties` or `fiftyoneDegreesHashInitManagerFromMemoryWithIndexedProperties` | `fiftyoneDegreesHashGetProfileIdsForPropertyAndValue`                                             | The number of profiles, with their ids set in an array the caller provides |
+| C                                 | As above                                                                                                                         | `fiftyoneDegreesHashIterateProfilesForPropertyAndValue`                                           | Each profile, passed to a callback                                         |
+| C++                               | `ConfigHash::setIndexedProperties`                                                                                               | `EngineHash::getProfileIdsForPropertyValue`                                                       | A `std::vector<uint32_t>` of profile ids                                   |
+| C#, Java, Node.js, Python and PHP | `setIndexedProperties` on `ConfigHashSwig`                                                                                       | `getProfileIdsForPropertyValue` on `EngineHashSwig`                                               | A `VectorUInt32Swig` of profile ids                                        |
+| Go and Rust                       | The C init functions, called through their bindings to the C API                                                                 | `fiftyoneDegreesHashGetProfileIdsForPropertyAndValue`, called through their bindings to the C API | The profile ids, as for C                                                  |
+
+The languages built on SWIG get these once their wrapper is generated again
+from the interface in 51degrees-cxx. In .NET the `NativeEngine` and
+`TacEngine` of `FiftyOne.DeviceDetection.PropertyKeyed` are On-premise
+Engines of this kind. Since
+[51degrees-dotnet#101](https://github.com/51Degrees/51degrees-dotnet/pull/101)
+they use the native lookup for a Property that the Device Detection
+on-premise Engine's builder names with
+`SetIndexedProperties("NativeModel", "TAC")`. For a Property the builder does
+not name, they still process every hardware profile at start-up to build an index
+of their own, which
+[51degrees-dotnet#30](https://github.com/51Degrees/51degrees-dotnet/issues/30)
+tracks. Java has no such Engine yet, which
+[51degrees-java#17](https://github.com/51Degrees/51degrees-java/issues/17)
+proposes.
+
+## Accepted Evidence
+
+| **Key**             | **Description**                                    |
+|---------------------|----------------------------------------------------|
+| `query.tac`         | The TAC to look up, as for the remote server       |
+| `query.nativemodel` | The native model name to look up                   |
+
+## Element Data
+
+| **Name** | **Type**                     | **Description**                                                                                                                                                                   |
+|----------|------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| profiles | `IReadOnlyList<IDeviceData>` | Every hardware profile holding the value, in the order of the data file. Each is interface compatible with the results from the cloud and on-premise Device Detection Engines. |
+
+The [JSON Builder](../../pipeline-specification/pipeline-elements/json-builder.md#process)
+MUST write `profiles` as an empty list however many profiles match, because
+the profiles are read on demand, so building the JSON reads no profile.
+
+## Start-up activity
+
+The Device Detection on-premise Engine whose data this Engine uses SHOULD be
+created with the NativeModel and TAC Properties named to index, so that the
+native index is built when its data set is initialized. An Engine of this
+kind MUST find the profiles through the native lookup, as the
+[profiles for a property value](device-detection-on-premise.md#profiles-for-a-property-value)
+require, rather than build an index of its own by processing every hardware
+profile at start-up.
+
+## Process
+
+1. Read the TAC or native model name from the Evidence.
+2. Get the ids of the profiles holding it from the native lookup, with the
+   Property name `TAC` or `NativeModel`.
+3. Add the ids to the Element Data, in the order returned, without reading
+   the profiles, so that a value many profiles hold costs only the lookup
+   until `profiles` is read. There is no limit on the number of ids.
+4. The first time `profiles` is read, populate an instance of the device
+   data for each id, in the order the ids were added, with the Property
+   values of that profile, for example by processing the id as
+   `51D_ProfileIds` Evidence with the Device Detection on-premise Engine.
+   A later read does not read the profiles again unless more ids have been
+   added since.
+
+## Refresh data
+
+When the data of the Device Detection on-premise Engine is refreshed, the
+native component indexes the same Properties again from the new data file.
+Nothing more is needed from this Engine.
+
+## Configuration options
+
+There are no configuration options associated with this Engine beyond those
+of the Device Detection on-premise Engine, where the Indexed Properties
+[configuration option](device-detection-on-premise.md#configuration-options)
+names the Properties to index.

device-detection-specification/pipeline-elements/uach-high-entropy-decoder.md

@@ -17,22 +17,35 @@ encoded using base-64 encoding. This element decodes and transforms this
 Evidence into values that match the HTTP header format so that it can
 be used by the Device Detection Engines.
 
+An on-premise Device Detection Engine whose native code does this
+conversion itself does not need this Element. See
+[Engines that convert the values](#engines-that-convert-the-values).
+
 ## Accepted Evidence
 
 - cookie.51d_gethighentropyvalues
 - query.51d_gethighentropyvalues
 
+The Device Detection Engines read `51D_StructuredUserAgent` in the same way,
+as `cookie.51d_structureduseragent` or `query.51d_structureduseragent`. Only the
+first value of either key that gives headers is used, and a `query` value is
+tried before any `cookie` value, as described in the [Process](#process)
+section.
+
 ## Element Data
 
-This Element is only implemented for
-[.NET](https://github.com/51Degrees/51degrees-dotnet/tree/main/device-detection/DeviceDetection)
-at time of writing.
-The reference implementation writes directly to Evidence, which will not
-be possible if Evidence is immutable.
-Instead, the Element can output its values using Element Data as described
-in the table below.
-The Device Detection Engines MUST be capable of using these values if
-available and falling back to Evidence values if they are not.
+The .NET and Java packages have no implementation of this Element, and the
+native code their on-premise Engines use converts the values itself.
+An implementation of this Element can write the decoded values to Evidence,
+which a Flow Element MAY do outside a
+[parallel group](../../pipeline-specification/features/thread-safety.md#evidence),
+as Evidence is not immutable.
+Alternatively, the Element can output its values using Element Data as
+described in the table below.
+A Device Detection Engine MUST be capable of using these values from
+Element Data if available, and falling back to Evidence values if they are
+not, unless it is an on-premise Engine whose native code converts the
+values itself.
 See [adding Evidence values](../../pipeline-specification/features/evidence.md#adding-evidence-values)
 for more details.
 
@@ -53,11 +66,43 @@ not needed for device detection, so are not featured here.
 
 - Get the encoded value from Evidence for the [relevant keys](#accepted-evidence)
   - Use the value with the `query` prefix if both are available
+    - A value is available when it is present and gives at least one header
+      when decoded, so a `query` value that cannot be decoded, or gives no
+      header, leaves the `cookie` value to be used
 - Decode the value from base 64
 - Convert the JSON representation into individual HTTP header values.
   - See [here](https://github.com/51Degrees/sua-uach-conversion/blob/main/src/convertSUAtoUACH.js)
     for an example of this using JavaScript.
 
+## Engines that convert the values
+
+The native code of the on-premise Device Detection Engine in
+[51degrees-cxx](https://github.com/51Degrees/51degrees-cxx) converts the
+`query.51d_gethighentropyvalues` and `cookie.51d_gethighentropyvalues`
+Evidence itself, before it matches, into header values that include those in
+the [Element Data](#element-data) table
+([hash.c lines 3902 to 3920](https://github.com/51Degrees/51degrees-cxx/blob/cfad8978827292b97ad08f306891edea09ece304/src/dd/hash/hash.c#L3902-L3920)
+and the list of headers in
+[transform.h lines 49 to 56](https://github.com/51Degrees/51degrees-cxx/blob/cfad8978827292b97ad08f306891edea09ece304/src/dd/transform.h#L49-L56)).
+It does so while special evidence processing is enabled, which is the default
+([config-dd.h line 90](https://github.com/51Degrees/51degrees-cxx/blob/cfad8978827292b97ad08f306891edea09ece304/src/dd/config-dd.h#L90)).
+
+An on-premise Device Detection Engine whose native code converts the values
+itself, and which accepts both keys and passes their values to that native
+code unchanged, need not read this Element's data. A Pipeline built around
+such an Engine does not need this Element.
+
+The native code converts only one such value, as
+[Accepted Evidence](#accepted-evidence) says. It tries every `query` value
+before any `cookie` value, and uses the first `51d_gethighentropyvalues` or
+`51d_structureduseragent` value that gives at least one header
+([hash.c lines 3949 to 3959](https://github.com/51Degrees/51degrees-cxx/blob/cfad8978827292b97ad08f306891edea09ece304/src/dd/hash/hash.c#L3949-L3959)
+and [lines 4287 to 4305](https://github.com/51Degrees/51degrees-cxx/blob/cfad8978827292b97ad08f306891edea09ece304/src/dd/hash/hash.c#L4287-L4305)).
+So where that first value is a `51d_structureduseragent` value, the high
+entropy values are not converted, and the header values come from the
+structured User-Agent alone, which gives no `sec-ch-ua` value
+([transformc.c lines 1105 to 1107](https://github.com/51Degrees/51degrees-cxx/blob/cfad8978827292b97ad08f306891edea09ece304/src/dd/transformc.c#L1105-L1107)).
+
 ## Configuration options
 
 This Element has no configuration options

device-detection-specification/required-examples.md

@@ -17,6 +17,11 @@ it is often difficult to carry out verification of the results produced by the
 example, and it is likely to be sufficient for a test to show that the example
 runs and completes without failure.
 
+Those tests MUST include a run of each example with no arguments or settings,
+as someone would run it from a fresh clone, so that the run uses the free data
+files. A run that is given a paid data file does not show that the example
+works out of the box.
+
 The need for examples to be runnable as an application and as a test suggests
 that they are written with example code to be executed as a public method, and
 a common pattern is established across examples.
@@ -73,23 +78,44 @@ to provide the same functionality.
 ### Test Data
 
 Many examples require test data consisting of HTTP Header Evidence - User Agent
-and UserAgent Client Hints. A YAML file containing 20,000 sets of such Evidence
-is distributed as one of the Git submodules referenced from on-premise Hash
-detection. It is also available in a repository in
-[GitHub](https://github.com/51Degrees/device-detection-data)
-where it is updated on a monthly basis.
+and UserAgent Client Hints. A YAML file containing 20,000 sets of such Evidence,
+`20000 Evidence Records.yml`, is linked from the `device-detection` folder of
+the [data repository](https://github.com/51Degrees/data), which each language
+repository holds as a submodule, and the data repository's `get-data-files`
+scripts download it.
 
 See [File References](#file-references) for a discussion of locating files.
 
 ### On-Premise Device Detection
 
-Reference implementation examples illustrate use of the Lite Data File that is
-distributed as one of the Git submodules referenced from on-premise Hash detection.
-It is also updated in [GitHub](https://github.com/51Degrees/device-detection-data)
-on a monthly basis.
+Examples MUST run out of the box with the free data files. Given no arguments
+or settings, an example uses the Lite Data File, `51Degrees-LiteV4.1.hash`,
+which the [data repository](https://github.com/51Degrees/data) links in its
+`device-detection` folder. An example looks for the file and, when it is
+missing, downloads it with the data repository's `get-data-files` script, so a
+fresh clone of a language repository runs every example with nothing set. A
+paid data file can be given as an argument or setting instead.
 
 See [File References](#file-references) for a discussion of locating files.
 
+Examples MUST also show what only the Enterprise Data File can do, because the
+examples are where a developer first sees what the paid data files add.
+Alongside Properties the Lite Data File holds, an example asks for Properties
+that only the Enterprise Data File holds, such as `HardwareVendor`,
+`HardwareName` and `DeviceType`. Where the data file in use has no value for
+one of them, the example says that the Property needs the Enterprise Data
+File, rather than leaving it out or showing a default value, and once in its
+output gives a link to the
+[51Degrees contact page](https://51degrees.com/contact-us?utm_source=github&utm_medium=docs&utm_campaign=specifications&utm_content=device-detection-specification-required-examples.md&utm_term=on-premise-device-detection)
+to ask for a copy. The link the example writes carries `utm_source=code`,
+`utm_medium=example`, the language repository's name as `utm_campaign` and
+the example's file as `utm_content`, so that an enquiry can be traced to the
+example that prompted it.
+
+An example whose purpose needs the Enterprise Data File or a License Key, such
+as one that updates the data file, explains that when it runs with the Lite
+Data File, gives the same link, and completes without failure.
+
 In the reference implementations it is stressed in comments as well as
 in the output of examples that
 the Lite data file has a small number of Properties.
@@ -241,6 +267,11 @@ using either TAC or native model name.
 These examples require the user to obtain a license key in order to
 configure an appropriate Resource Key.
 
+An implementation that provides the
+[hardware profile lookup on-premise](pipeline-elements/hardware-profile-lookup-on-premise.md)
+MAY also illustrate it, with an Enterprise data file for the native model
+name and a TAC data file for TAC.
+
 See documentation
 [TAC](https://51degrees.com/documentation/_examples__device_detection__tac_lookup__cloud.html?utm_source=github&utm_medium=docs&utm_campaign=specifications&utm_content=device-detection-specification-required-examples.md&utm_term=tac-native-key-lookup),
 [Native key](https://51degrees.com/documentation/_examples__device_detection__native_key_lookup__cloud.html?utm_source=github&utm_medium=docs&utm_campaign=specifications&utm_content=device-detection-specification-required-examples.md&utm_term=tac-native-key-lookup)