diff --git a/docs/whpg/6x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md b/docs/whpg/6x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md index 19437f4..ef8c5f9 100644 --- a/docs/whpg/6x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md +++ b/docs/whpg/6x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md @@ -13,6 +13,6 @@ Creates a readable external table, *ext_expenses,* using the `pxf` protocol. The ``` -Refer to [Accessing External Data with PXF](../../pxf-overview.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System. +Refer to [HDFS](/pxf/6x/connecting/hadoop/hdfs.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System. **Parent topic:** [Examples for Creating External Tables](index.md) diff --git a/docs/whpg/6x/admin_guide/external/external-tables/index.md b/docs/whpg/6x/admin_guide/external/external-tables/index.md index eb38c29..01f62a2 100644 --- a/docs/whpg/6x/admin_guide/external/external-tables/index.md +++ b/docs/whpg/6x/admin_guide/external/external-tables/index.md @@ -29,7 +29,7 @@ Use one of the following protocols to access external table data sources. You ca The `pxf://` and `s3://` protocols are custom data access protocols, where the `file://`, `gpfdist://`, and `gpfdists://` protocols are implemented internally in WarehousePG. The custom and internal protocols differ in these ways: -- `pxf://` and `s3://` are custom protocols that must be registered using the `CREATE EXTENSION` command (`pxf`) or the `CREATE PROTOCOL` command (`s3`). Registering the PXF extension in a database creates the `pxf` protocol. (See [Accessing External Data with PXF](../pxf-overview.md).) To use the `s3` protocol, you must configure the database and register the `s3` protocol. (See [Configuring the s3 Protocol](s3-protocol.md#s3_prereq).) Internal protocols are always present and cannot be unregistered. +- `pxf://` and `s3://` are custom protocols that must be registered using the `CREATE EXTENSION` command (`pxf`) or the `CREATE PROTOCOL` command (`s3`). Registering the PXF extension in a database creates the `pxf` protocol. (See [PXF for WarehousePG](/pxf/6x/index.md).) To use the `s3` protocol, you must configure the database and register the `s3` protocol. (See [Configuring the s3 Protocol](s3-protocol.md#s3_prereq).) Internal protocols are always present and cannot be unregistered. - When a custom protocol is registered, a row is added to the `pg_extprotocol` catalog table to specify the handler functions that implement the protocol. The protocol's shared libraries must have been installed on all WarehousePG hosts. The internal protocols are not represented in the `pg_extprotocol` table and have no additional libraries to install. - To grant users permissions on custom protocols, you use `GRANT [SELECT | INSERT | ALL] ON PROTOCOL`. To allow (or deny) users permissions on the internal protocols, you use `CREATE ROLE` or `ALTER ROLE` to add the `CREATEEXTTABLE` (or `NOCREATEEXTTABLE`) attribute to each user's role. diff --git a/docs/whpg/6x/admin_guide/external/external-tables/pxf-protocol.md b/docs/whpg/6x/admin_guide/external/external-tables/pxf-protocol.md index 41b85f9..a95a35e 100644 --- a/docs/whpg/6x/admin_guide/external/external-tables/pxf-protocol.md +++ b/docs/whpg/6x/admin_guide/external/external-tables/pxf-protocol.md @@ -11,6 +11,6 @@ When you use the `pxf` protocol to query an external data store, you specify the You must explicitly initialize and start PXF before you can use the `pxf` protocol to read or write external data. You must also enable PXF in each database in which you want to allow users to create external tables to access external data, and grant permissions on the `pxf` protocol to those WarehousePG users. -For detailed information about configuring and using PXF and the `pxf` protocol, refer to [Accessing External Data with PXF](../pxf-overview.md). +For detailed information about configuring and using PXF and the `pxf` protocol, refer to [PXF for WarehousePG](/pxf/6x/index.md). **Parent topic:** [Defining External Tables](index.md) diff --git a/docs/whpg/6x/admin_guide/external/index.md b/docs/whpg/6x/admin_guide/external/index.md index 5efc441..6aafd45 100644 --- a/docs/whpg/6x/admin_guide/external/index.md +++ b/docs/whpg/6x/admin_guide/external/index.md @@ -3,7 +3,6 @@ title: Working with External Data navigation: - foreign - external-tables - - pxf-overview - using-gpfdist --- @@ -12,13 +11,13 @@ Both external and foreign tables provide access to data stored in data sources o An external table is a WarehousePG table backed with data that resides outside of the database. You create a readable external table to read data from the external data source and create a writable external table to write data to the external source. You can use external tables in SQL commands just as you would a regular database table. For example, you can `SELECT` (readable external table), `INSERT` (writable external table), and join external tables with other WarehousePG tables. External tables are most often used to load and unload database data. Refer to [Defining External Tables](external-tables/index.md) for more information about using external tables to access external data. -[Accessing External Data with PXF](pxf-overview.md) describes using PXF and external tables to access external data sources. +[PXF for WarehousePG](/pxf/6x/index.md) describes using PXF and external tables to access external data sources. A foreign table is a different kind of WarehousePG table backed with data that resides outside of the database. You can both read from and write to the same foreign table. You can similarly use foreign tables in SQL commands as described above for external tables. Refer to [Accessing External Data with Foreign Tables](foreign/index.md) for more information about accessing external data using foreign tables. Web-based external tables provide access to data served by an HTTP server or an operating system process. See [Creating and Using External Web Tables](external-tables/creating-and-using-web-external-tables.md) for more about web-based tables. -- **[Accessing External Data with PXF](pxf-overview.md)** +- **[PXF for WarehousePG](/pxf/6x/index.md)** Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. - **[Defining External Tables](external-tables/index.md)** diff --git a/docs/whpg/6x/admin_guide/external/pxf-overview.md b/docs/whpg/6x/admin_guide/external/pxf-overview.md deleted file mode 100644 index a096eb1..0000000 --- a/docs/whpg/6x/admin_guide/external/pxf-overview.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: Accessing External Data with PXF - ---- - -Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. - -PXF is installed with Hadoop and Object Storage connectors. These connectors enable you to read external data stored in text, Avro, JSON, RCFile, Parquet, SequenceFile, and ORC formats. You can use the JDBC connector to access an external SQL database. - -The WarehousePG Platform Extension Framework includes a C-language extension and a Java service. After you configure and initialize PXF, you start a single PXF JVM process on each WarehousePG segment host. This long- running process concurrently serves multiple query requests. - -For detailed information about the architecture of and using PXF, refer to the PXF documentation. - -**Parent topic:** [Working with External Data](index.md) - -**Parent topic:** [Loading and Unloading Data](../load/index.md) diff --git a/docs/whpg/6x/admin_guide/load/index.md b/docs/whpg/6x/admin_guide/load/index.md index 86b13f0..ebcd7dd 100644 --- a/docs/whpg/6x/admin_guide/load/index.md +++ b/docs/whpg/6x/admin_guide/load/index.md @@ -26,7 +26,7 @@ WarehousePG can read from and write to several types of external data sources, i - External web tables provide access to dynamic data. They can be backed with data from URLs accessed using the HTTP protocol or by the output of an OS script running on one or more segments. - The `gpfdist` utility is the WarehousePG parallel file distribution program. It is an HTTP server that is used with external tables to allow WarehousePG segments to load external data in parallel, from multiple file systems. You can run multiple instances of `gpfdist` on different hosts and network interfaces and access them in parallel. - The `gpload` utility automates the steps of a load task using `gpfdist` and a YAML-formatted control file. -- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [Accessing External Data with PXF](../external/pxf-overview.md). +- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md). The method you choose to load data depends on the characteristics of the source data—its location, size, format, and any transformations required. diff --git a/docs/whpg/6x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md b/docs/whpg/6x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md index 91e9f1c..f415fa9 100644 --- a/docs/whpg/6x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md +++ b/docs/whpg/6x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md @@ -8,7 +8,7 @@ navigation: Writable external tables that output data to files can use the WarehousePG parallel file server program, gpfdist, or the WarehousePG Platform Extension Framework (PXF), WarehousePG's interface to Hadoop. -Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [Accessing External Data with PXF](../../../external/pxf-overview.md) for instructions on setting up PXF for use with an external table +Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [PXF for WarehousePG](/pxf/6x/index.md) for instructions on setting up PXF for use with an external table - With a writable external table using the gpfdist protocol, the WarehousePG segments send their data to gpfdist, which writes the data to the named file. gpfdist must run on a host that the WarehousePG segments can access over the network. gpfdist points to a file location on the output host and writes data received from the WarehousePG segments to the file. To divide the output data among multiple files, list multiple gpfdist URIs in your writable external table definition. - A writable external web table sends data to an application as a stream of data. For example, unload data from WarehousePG and send it to an application that connects to another database or ETL tool to load the data elsewhere. Writable external web tables use the `EXECUTE` clause to specify a shell command, script, or application to run on the segment hosts and accept an input stream of data. See [Defining a Command-Based Writable External Web Table](../defining-a-command-based-writable-external-web-table/index.md) for more information about using `EXECUTE` commands in a writable external table definition. diff --git a/docs/whpg/6x/admin_guide/managing/compression.md b/docs/whpg/6x/admin_guide/managing/compression.md index b83b8ee..d4f6c6e 100644 --- a/docs/whpg/6x/admin_guide/managing/compression.md +++ b/docs/whpg/6x/admin_guide/managing/compression.md @@ -9,7 +9,7 @@ You can configure support for data compression with these features and utilities - Append-optimized tables support compressing table data. See [CREATE TABLE](../../ref_guide/sql_commands/CREATE_TABLE.md). - User-defined data types can be defined to compress data. See [CREATE TYPE](../../ref_guide/sql_commands/CREATE_TYPE.md). -- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](../external/pxf-overview.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md). +- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](/pxf/6x/index.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md). - Workfiles (temporary spill files that are created when running a query that requires more memory than it is allocated) can be compressed. See the server configuration parameter [gp_workfile_compression](../../ref_guide/config_params/guc-list.md). - The WarehousePG utilities gpbackup, gprestore, [gpload](../../ref_guide/utility_guide/reference/gpload.md), and [gplogfilter](../../ref_guide/utility_guide/reference/gplogfilter.md) support compression. diff --git a/docs/whpg/6x/install_guide/additional_modules/external_components/index.md b/docs/whpg/6x/install_guide/additional_modules/external_components/index.md index c262917..21d6637 100644 --- a/docs/whpg/6x/install_guide/additional_modules/external_components/index.md +++ b/docs/whpg/6x/install_guide/additional_modules/external_components/index.md @@ -6,6 +6,6 @@ description: Installation guides for tools that integrate with WarehousePG but a These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`. -- [WarehousePG Platform Extension Framework (PXF)](../../data_sci_pkgs/install_pxf.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop. -- whpg-backup - Parallel backup and restore for WarehousePG clusters. -- whpg-backup-s3-plugin - Amazon S3 support for whpg-backup. +- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop. +- [whpg-backup](/whpg-backup/index.md) - Parallel backup and restore for WarehousePG clusters. +- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Amazon S3 support for whpg-backup. diff --git a/docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md b/docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md index 88788c1..47983f7 100644 --- a/docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md +++ b/docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md @@ -5,6 +5,6 @@ title: Platform Extension Framework (PXF) *Optional.* If you do not plan to use PXF, no action is necessary. -If you plan to use PXF, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for introductory PXF information. +If you plan to use PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md) for installation and usage information. **Parent topic:** [Installing Optional Extensions (WarehousePG)](index.md) diff --git a/docs/whpg/6x/install_guide/platform-requirements.md b/docs/whpg/6x/install_guide/platform-requirements.md index c251a9d..6927b7b 100644 --- a/docs/whpg/6x/install_guide/platform-requirements.md +++ b/docs/whpg/6x/install_guide/platform-requirements.md @@ -191,7 +191,7 @@ These WarehousePG extensions are installed with WarehousePG ### Data Connectors -- WarehousePG Platform Extension Framework (PXF) - PXF provides access to Hadoop, object store, and SQL external data stores. Refer to [Accessing External Data with PXF](../admin_guide/external/pxf-overview.md) in the *WarehousePG Administrator Guide* for PXF configuration and usage information. +- WarehousePG Platform Extension Framework (PXF) - PXF provides access to Hadoop, object store, and SQL external data stores. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for PXF configuration and usage information. - WarehousePG Connector for Apache Spark v1.6.2 - The WarehousePG Connector for Apache Spark supports high speed, parallel data transfer between WarehousePG and an Apache Spark cluster using Spark’s Scala API. diff --git a/docs/whpg/6x/ref_guide/modules/external_components/index.md b/docs/whpg/6x/ref_guide/modules/external_components/index.md index 0d95a2b..5dadc92 100644 --- a/docs/whpg/6x/ref_guide/modules/external_components/index.md +++ b/docs/whpg/6x/ref_guide/modules/external_components/index.md @@ -6,6 +6,6 @@ description: Separately installed tools and frameworks that integrate with Wareh These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`. -- [WarehousePG Platform Extension Framework (PXF)](../../../install_guide/data_sci_pkgs/install_pxf.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop. -- whpg-backup - Provides parallel backup and restore for WarehousePG clusters. -- whpg-backup-s3-plugin - Adds Amazon S3 support to whpg-backup. +- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop. +- [whpg-backup](/whpg-backup/index.md) - Provides parallel backup and restore for WarehousePG clusters. +- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Adds Amazon S3 support to whpg-backup. diff --git a/docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md b/docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md index 8321f7a..24ad530 100644 --- a/docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md +++ b/docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md @@ -184,7 +184,7 @@ See [Working with External Data](../../admin_guide/external/index.md) for detail - LOCATION ('protocol://\[host\[:port]]/path/file' \[, ...]) - If you use the `pxf` protocol to access an external data source, refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about the `pxf` protocol. + If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about the `pxf` protocol. If you use the `s3` protocol to read or write to S3, refer to [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for additional information about the `s3` protocol `LOCATION` clause syntax. @@ -240,13 +240,13 @@ See [Working with External Data](../../admin_guide/external/index.md) for detail When the `FORMAT` clause identfies delimited text (`TEXT`) or comma separated values (`CSV`) format, formatting options are similar to those available with the PostgreSQL [COPY](COPY.md) command. If the data in the file does not use the default column delimiter, escape character, null string and so on, you must specify the additional formatting options so that the data in the external file is read correctly by WarehousePG. For information about using a custom format, see "Loading and Unloading Data" in the *WarehousePG Administrator Guide*. - If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF. + If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF. - FORMAT 'CUSTOM' (formatter=formatter_specification) Specifies a custom data format. The formatter_specification specifies the function to use to format the data, followed by comma-separated parameters to the formatter function. The length of the formatter specification, the string including `Formatter=`, can be up to approximately 50K bytes. - If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF. + If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF. For general information about using a custom format, see "Loading and Unloading Data" in the *WarehousePG Administrator Guide*. diff --git a/docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md b/docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md index 29d6eee..30801bf 100644 --- a/docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md +++ b/docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md @@ -67,7 +67,7 @@ For information about creating and enabling a custom data access protocol, see " ## Notes -WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md#configuring-the-s3-protocol) for information about enabling the `S3` protocol. Refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about using the `pxf` protocol. +WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md#configuring-the-s3-protocol) for information about enabling the `S3` protocol. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using the `pxf` protocol. Any shared library that implements a data access protocol must be located in the same location on all WarehousePG segment hosts. For example, the shared library can be in a location specified by the operating system environment variable `LD_LIBRARY_PATH` on all hosts. You can also specify the location when you define the handler function. For example, when you define the `s3` protocol in the `CREATE PROTOCOL` command, you specify `$libdir/gps3ext.so` as the location of the shared object, where `$libdir` is located at `$GPHOME/lib`. diff --git a/docs/whpg/6x/security-guide/SecuringGPDB.md b/docs/whpg/6x/security-guide/SecuringGPDB.md index bc10c5a..a4ce28b 100644 --- a/docs/whpg/6x/security-guide/SecuringGPDB.md +++ b/docs/whpg/6x/security-guide/SecuringGPDB.md @@ -24,7 +24,7 @@ The essential security requirements fall into the following categories: ## Accessing a Kerberized Hadoop Cluster -You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to Configuring PXF for Secure HDFS for the procedure to perform this setup. +You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to [Authenticating with Kerberos](/pxf/6x/connecting/hadoop/kerberos.md) for the procedure to perform this setup. diff --git a/docs/whpg/7x/admin_guide/analytics/index.md b/docs/whpg/7x/admin_guide/analytics/index.md index e34cd14..49a95e4 100644 --- a/docs/whpg/7x/admin_guide/analytics/index.md +++ b/docs/whpg/7x/admin_guide/analytics/index.md @@ -65,7 +65,7 @@ The WarehousePG analytics capabilities allow you to: - Leverage existing SQL knowledge: WarehousePG can run dozens of statistical, machine learning, and graph methods, via SQL. - Train more models in less time by taking advantage of the parallelism in the MPP architecture and in-database analytics. - Access the data where it lives, therefore integrate data and analytics in one place. WarehousePG is infrastructure-agnostic and runs on bare metal, private cloud, and public cloud deployments. -- Use a multitude of data extensions. WarehousePG supports extensions for HDFS, Hive, and HBase as well as reading/writing data from/to cloud storage, including Amazon S3 objects. Review the capabilities of the WarehousePG [Platform Extension Framework (PXF)](../external/pxf-overview.md), which provides *connectors* that enable you to access data stored in sources external to your WarehousePG deployment. +- Use a multitude of data extensions. WarehousePG supports extensions for HDFS, Hive, and HBase as well as reading/writing data from/to cloud storage, including Amazon S3 objects. Review the capabilities of the WarehousePG [Platform Extension Framework (PXF)](/pxf/6x/index.md), which provides *connectors* that enable you to access data stored in sources external to your WarehousePG deployment. - Use familiar and leading BI and advanced analytics software that are ODBC/JDBC compatible, or have native integrations, including SAS, IBM Cognos, SAP Analytics Solutions, Qlik, Tableau, Apache Zeppelin, and Jupyter. - Run deep learning algorithms using popular frameworks like Keras and TensorFlow in an MPP relational database, with GPU (Graphical Processing Unit) acceleration. - Use containers capable of isolating executors from the host OS. WarehousePG PL/Container implements a trusted language execution engine which permits customized data science workloads or environments created for different end user workloads. diff --git a/docs/whpg/7x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md b/docs/whpg/7x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md index 19437f4..ef8c5f9 100644 --- a/docs/whpg/7x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md +++ b/docs/whpg/7x/admin_guide/external/external-tables/creating-external-tables---examples/example-5-text-format-on-a-hadoop-distributed-file-server.md @@ -13,6 +13,6 @@ Creates a readable external table, *ext_expenses,* using the `pxf` protocol. The ``` -Refer to [Accessing External Data with PXF](../../pxf-overview.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System. +Refer to [HDFS](/pxf/6x/connecting/hadoop/hdfs.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System. **Parent topic:** [Examples for Creating External Tables](index.md) diff --git a/docs/whpg/7x/admin_guide/external/external-tables/pxf-protocol.md b/docs/whpg/7x/admin_guide/external/external-tables/pxf-protocol.md index 41b85f9..a95a35e 100644 --- a/docs/whpg/7x/admin_guide/external/external-tables/pxf-protocol.md +++ b/docs/whpg/7x/admin_guide/external/external-tables/pxf-protocol.md @@ -11,6 +11,6 @@ When you use the `pxf` protocol to query an external data store, you specify the You must explicitly initialize and start PXF before you can use the `pxf` protocol to read or write external data. You must also enable PXF in each database in which you want to allow users to create external tables to access external data, and grant permissions on the `pxf` protocol to those WarehousePG users. -For detailed information about configuring and using PXF and the `pxf` protocol, refer to [Accessing External Data with PXF](../pxf-overview.md). +For detailed information about configuring and using PXF and the `pxf` protocol, refer to [PXF for WarehousePG](/pxf/6x/index.md). **Parent topic:** [Defining External Tables](index.md) diff --git a/docs/whpg/7x/admin_guide/external/foreign/index.md b/docs/whpg/7x/admin_guide/external/foreign/index.md index 8b08cbd..9c2e479 100644 --- a/docs/whpg/7x/admin_guide/external/foreign/index.md +++ b/docs/whpg/7x/admin_guide/external/foreign/index.md @@ -9,7 +9,7 @@ WarehousePG implements portions of the SQL/MED specification, allowing you to ac You can access foreign data with help from a *foreign-data wrapper*. A foreign-data wrapper is a library that communicates with a remote data source. This library hides the source-specific connection and data access details. -The WarehousePG distribution includes the [postgres_fdw](../../../ref_guide/modules/bundled/postgres_fdw.md) foreign data wrapper. +The WarehousePG distribution includes the [postgres_fdw](../../../ref_guide/modules/bundled/postgres_fdw.md) foreign data wrapper. If you use PXF to access external data, the [pxf_fdw](/pxf/6x/foreign-data-wrapper.md) foreign data wrapper lets you query those same PXF-connected sources through standard foreign tables instead of `pxf://` external tables. If none of the existing PostgreSQL or WarehousePG foreign-data wrappers suit your needs, you can write your own as described in [Writing a Foreign Data Wrapper](devel-fdw.md). diff --git a/docs/whpg/7x/admin_guide/external/index.md b/docs/whpg/7x/admin_guide/external/index.md index 55b886e..8fb8a46 100644 --- a/docs/whpg/7x/admin_guide/external/index.md +++ b/docs/whpg/7x/admin_guide/external/index.md @@ -3,7 +3,6 @@ title: Working with External Data navigation: - foreign - external-tables - - pxf-overview - using-gpfdist redirects: - working-with-file-based-ext-tables @@ -48,4 +47,4 @@ Web-based external tables provide access to data served by an HTTP server or an ## About Accessing External Data with pxf -Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. Refer to [Accessing External Data with PXF](pxf-overview.md) for more information about using PXF. +Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for more information about using PXF. diff --git a/docs/whpg/7x/admin_guide/external/pxf-overview.md b/docs/whpg/7x/admin_guide/external/pxf-overview.md deleted file mode 100644 index a586bd9..0000000 --- a/docs/whpg/7x/admin_guide/external/pxf-overview.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -title: Accessing External Data with PXF - ---- - -Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition. - -PXF is installed with Hadoop and Object Storage connectors. These connectors enable you to read and write external data stored in text, Avro, JSON, RCFile, Parquet, SequenceFile, and ORC formats. You can use the JDBC connector to access an external SQL database. - -The WarehousePG Platform Extension Framework includes a C-language extension and a Java service. After you configure and initialize PXF, you start a single PXF JVM process on each WarehousePG segment host. This long- running process concurrently serves multiple query requests. - -For detailed information about the architecture of and using PXF, refer to the WarehousePG Platform Extension Framework PXF documentation - -**Parent topic:** [Working with External Data](index.md) diff --git a/docs/whpg/7x/admin_guide/load/index.md b/docs/whpg/7x/admin_guide/load/index.md index 562afdb..319e17e 100644 --- a/docs/whpg/7x/admin_guide/load/index.md +++ b/docs/whpg/7x/admin_guide/load/index.md @@ -28,7 +28,7 @@ WarehousePG can read from and write to several types of external data sources, i - External web tables provide access to dynamic data. They can be backed with data from URLs accessed using the HTTP protocol or by the output of an OS script running on one or more segments. - The `gpfdist` utility is the WarehousePG parallel file distribution program. It is an HTTP server that is used with external tables to allow WarehousePG segments to load external data in parallel, from multiple file systems. You can run multiple instances of `gpfdist` on different hosts and network interfaces and access them in parallel. - The `gpload` utility automates the steps of a load task using `gpfdist` and a YAML-formatted control file. -- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [Accessing External Data with PXF](../external/pxf-overview.md). +- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md). The method you choose to load data depends on the characteristics of the source data—its location, size, format, and any transformations required. diff --git a/docs/whpg/7x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md b/docs/whpg/7x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md index 91e9f1c..f415fa9 100644 --- a/docs/whpg/7x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md +++ b/docs/whpg/7x/admin_guide/load/unloading-data-from-warehousepg/defining-a-file-based-writable-external-table/index.md @@ -8,7 +8,7 @@ navigation: Writable external tables that output data to files can use the WarehousePG parallel file server program, gpfdist, or the WarehousePG Platform Extension Framework (PXF), WarehousePG's interface to Hadoop. -Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [Accessing External Data with PXF](../../../external/pxf-overview.md) for instructions on setting up PXF for use with an external table +Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [PXF for WarehousePG](/pxf/6x/index.md) for instructions on setting up PXF for use with an external table - With a writable external table using the gpfdist protocol, the WarehousePG segments send their data to gpfdist, which writes the data to the named file. gpfdist must run on a host that the WarehousePG segments can access over the network. gpfdist points to a file location on the output host and writes data received from the WarehousePG segments to the file. To divide the output data among multiple files, list multiple gpfdist URIs in your writable external table definition. - A writable external web table sends data to an application as a stream of data. For example, unload data from WarehousePG and send it to an application that connects to another database or ETL tool to load the data elsewhere. Writable external web tables use the `EXECUTE` clause to specify a shell command, script, or application to run on the segment hosts and accept an input stream of data. See [Defining a Command-Based Writable External Web Table](../defining-a-command-based-writable-external-web-table/index.md) for more information about using `EXECUTE` commands in a writable external table definition. diff --git a/docs/whpg/7x/admin_guide/managing/compression.md b/docs/whpg/7x/admin_guide/managing/compression.md index e230b1b..b306388 100644 --- a/docs/whpg/7x/admin_guide/managing/compression.md +++ b/docs/whpg/7x/admin_guide/managing/compression.md @@ -9,7 +9,7 @@ You can configure support for data compression with these features and utilities - Append-optimized tables support compressing table data. See [CREATE TABLE](../../ref_guide/sql_commands/CREATE_TABLE.md). - User-defined data types can be defined to compress data. See [CREATE TYPE](../../ref_guide/sql_commands/CREATE_TYPE.md). -- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](../external/pxf-overview.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md). +- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](/pxf/6x/index.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md). - Workfiles (temporary spill files that are created when running a query that requires more memory than it is allocated) can be compressed. See the server configuration parameter [gp_workfile_compression](../../ref_guide/config_params/guc-list.md). - The WarehousePG utilities `gpbackup`, `gprestore`, `gpload`, `gplogfilter` support compression. diff --git a/docs/whpg/7x/install_guide/additional_modules/external_components/index.md b/docs/whpg/7x/install_guide/additional_modules/external_components/index.md index 7c5d39d..1b7ab66 100644 --- a/docs/whpg/7x/install_guide/additional_modules/external_components/index.md +++ b/docs/whpg/7x/install_guide/additional_modules/external_components/index.md @@ -8,6 +8,6 @@ navigation: These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`. -- [WarehousePG Platform Extension Framework (PXF)](install_pxf.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop. -- whpg-backup - Parallel backup and restore for WarehousePG clusters. -- whpg-backup-s3-plugin - Amazon S3 support for whpg-backup. +- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop. +- [whpg-backup](/whpg-backup/index.md) - Parallel backup and restore for WarehousePG clusters. +- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Amazon S3 support for whpg-backup. diff --git a/docs/whpg/7x/install_guide/additional_modules/external_components/install_pxf.md b/docs/whpg/7x/install_guide/additional_modules/external_components/install_pxf.md index 9f0db6f..fe15024 100644 --- a/docs/whpg/7x/install_guide/additional_modules/external_components/install_pxf.md +++ b/docs/whpg/7x/install_guide/additional_modules/external_components/install_pxf.md @@ -5,6 +5,6 @@ title: WarehousePG Platform Extension Framework (PXF) *Optional.* If you do not plan to use PXF, no action is necessary. -If you plan to use PXF, refer to [Accessing External Data with PXF](../../../admin_guide/external/pxf-overview.md) for introductory PXF information. +If you plan to use PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md) for installation and usage information. **Parent topic:** [Installing Optional Extensions (WarehousePG)](index.md) diff --git a/docs/whpg/7x/ref_guide/modules/external_components/index.md b/docs/whpg/7x/ref_guide/modules/external_components/index.md index ee96d3f..aecfa4a 100644 --- a/docs/whpg/7x/ref_guide/modules/external_components/index.md +++ b/docs/whpg/7x/ref_guide/modules/external_components/index.md @@ -6,6 +6,6 @@ description: Separately installed tools and frameworks that integrate with Wareh These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`. -- [WarehousePG Platform Extension Framework (PXF)](../../../install_guide/additional_modules/external_components/install_pxf.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop. -- whpg-backup - Provides parallel backup and restore for WarehousePG clusters. -- whpg-backup-s3-plugin - Adds Amazon S3 support to whpg-backup. +- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop. +- [whpg-backup](/whpg-backup/index.md) - Provides parallel backup and restore for WarehousePG clusters. +- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Adds Amazon S3 support to whpg-backup. diff --git a/docs/whpg/7x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md b/docs/whpg/7x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md index 23703be..33e380c 100644 --- a/docs/whpg/7x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md +++ b/docs/whpg/7x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md @@ -180,7 +180,7 @@ The data type of the column. ##### LOCATION ('protocol://\[host\[:port]]/path/file' \[, ...]) -If you use the `pxf` protocol to access an external data source, refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about the `pxf` protocol. +If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about the `pxf` protocol. If you use the `s3` protocol to read or write to S3, refer to [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for additional information about the `s3` protocol `LOCATION` clause syntax. @@ -233,13 +233,13 @@ For writable external tables, the command specified in the `EXECUTE` clause must When the `FORMAT` clause identfies delimited text (`TEXT`) or comma separated values (`CSV`) format, formatting options are similar to those available with the PostgreSQL [COPY](COPY.md) command. If the data in the file does not use the default column delimiter, escape character, null string and so on, you must specify the additional formatting options so that the data in the external file is read correctly by WarehousePG. For information about using a custom format, see [Loading and Unloading Data](../../admin_guide/load/index.md) in the *WarehousePG Administrator Guide*. -If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF. +If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF. ##### FORMAT 'CUSTOM' (formatter=formatter_specification) Specifies a custom data format. The formatter_specification specifies the function to use to format the data, followed by comma-separated parameters to the formatter function. The length of the formatter specification, the string including `Formatter=`, can be up to approximately 50K bytes. -If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF. +If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF. For general information about using a custom format, see "Loading and Unloading Data" in the *WarehousePG Administrator Guide*. diff --git a/docs/whpg/7x/ref_guide/sql_commands/CREATE_PROTOCOL.md b/docs/whpg/7x/ref_guide/sql_commands/CREATE_PROTOCOL.md index 9a702fa..bff695a 100644 --- a/docs/whpg/7x/ref_guide/sql_commands/CREATE_PROTOCOL.md +++ b/docs/whpg/7x/ref_guide/sql_commands/CREATE_PROTOCOL.md @@ -52,7 +52,7 @@ An optional validator function that validates the URL specified in the `CREATE E ## Notes -WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for information about enabling the `S3` protocol. Refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about using the `pxf` protocol. +WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for information about enabling the `S3` protocol. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using the `pxf` protocol. Any shared library that implements a data access protocol must be located in the same location on all WarehousePG segment hosts. For example, the shared library can be in a location specified by the operating system environment variable `LD_LIBRARY_PATH` on all hosts. You can also specify the location when you define the handler function. For example, when you define the `s3` protocol in the `CREATE PROTOCOL` command, you specify `$libdir/gps3ext.so` as the location of the shared object, where `$libdir` is located at `$GPHOME/lib`. diff --git a/docs/whpg/7x/security_guide/securing_whpg.md b/docs/whpg/7x/security_guide/securing_whpg.md index dfb5906..b70d776 100644 --- a/docs/whpg/7x/security_guide/securing_whpg.md +++ b/docs/whpg/7x/security_guide/securing_whpg.md @@ -18,7 +18,7 @@ The essential security requirements fall into the following categories: ## Accessing a Kerberized Hadoop Cluster -You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to Configuring PXF for Secure HDFS for the procedure to perform this setup. +You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to [Authenticating with Kerberos](/pxf/6x/connecting/hadoop/kerberos.md) for the procedure to perform this setup.