Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,6 @@ Creates a readable external table, *ext_expenses,* using the `pxf` protocol. The

```

Refer to [Accessing External Data with PXF](../../pxf-overview.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System.
Refer to [HDFS](/pxf/6x/connecting/hadoop/hdfs.md) for information about using the WarehousePG Platform Extension Framework (PXF) to access data on a Hadoop Distributed File System.

**Parent topic:** [Examples for Creating External Tables](index.md)
2 changes: 1 addition & 1 deletion docs/whpg/6x/admin_guide/external/external-tables/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ Use one of the following protocols to access external table data sources. You ca

The `pxf://` and `s3://` protocols are custom data access protocols, where the `file://`, `gpfdist://`, and `gpfdists://` protocols are implemented internally in WarehousePG. The custom and internal protocols differ in these ways:

- `pxf://` and `s3://` are custom protocols that must be registered using the `CREATE EXTENSION` command (`pxf`) or the `CREATE PROTOCOL` command (`s3`). Registering the PXF extension in a database creates the `pxf` protocol. (See [Accessing External Data with PXF](../pxf-overview.md).) To use the `s3` protocol, you must configure the database and register the `s3` protocol. (See [Configuring the s3 Protocol](s3-protocol.md#s3_prereq).) Internal protocols are always present and cannot be unregistered.
- `pxf://` and `s3://` are custom protocols that must be registered using the `CREATE EXTENSION` command (`pxf`) or the `CREATE PROTOCOL` command (`s3`). Registering the PXF extension in a database creates the `pxf` protocol. (See [PXF for WarehousePG](/pxf/6x/index.md).) To use the `s3` protocol, you must configure the database and register the `s3` protocol. (See [Configuring the s3 Protocol](s3-protocol.md#s3_prereq).) Internal protocols are always present and cannot be unregistered.
- When a custom protocol is registered, a row is added to the `pg_extprotocol` catalog table to specify the handler functions that implement the protocol. The protocol's shared libraries must have been installed on all WarehousePG hosts. The internal protocols are not represented in the `pg_extprotocol` table and have no additional libraries to install.
- To grant users permissions on custom protocols, you use `GRANT [SELECT | INSERT | ALL] ON PROTOCOL`. To allow (or deny) users permissions on the internal protocols, you use `CREATE ROLE` or `ALTER ROLE` to add the `CREATEEXTTABLE` (or `NOCREATEEXTTABLE`) attribute to each user's role.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,6 @@ When you use the `pxf` protocol to query an external data store, you specify the

You must explicitly initialize and start PXF before you can use the `pxf` protocol to read or write external data. You must also enable PXF in each database in which you want to allow users to create external tables to access external data, and grant permissions on the `pxf` protocol to those WarehousePG users.

For detailed information about configuring and using PXF and the `pxf` protocol, refer to [Accessing External Data with PXF](../pxf-overview.md).
For detailed information about configuring and using PXF and the `pxf` protocol, refer to [PXF for WarehousePG](/pxf/6x/index.md).

**Parent topic:** [Defining External Tables](index.md)
5 changes: 2 additions & 3 deletions docs/whpg/6x/admin_guide/external/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ title: Working with External Data
navigation:
- foreign
- external-tables
- pxf-overview
- using-gpfdist

---
Expand All @@ -12,13 +11,13 @@ Both external and foreign tables provide access to data stored in data sources o

An external table is a WarehousePG table backed with data that resides outside of the database. You create a readable external table to read data from the external data source and create a writable external table to write data to the external source. You can use external tables in SQL commands just as you would a regular database table. For example, you can `SELECT` (readable external table), `INSERT` (writable external table), and join external tables with other WarehousePG tables. External tables are most often used to load and unload database data. Refer to [Defining External Tables](external-tables/index.md) for more information about using external tables to access external data.

[Accessing External Data with PXF](pxf-overview.md) describes using PXF and external tables to access external data sources.
[PXF for WarehousePG](/pxf/6x/index.md) describes using PXF and external tables to access external data sources.

A foreign table is a different kind of WarehousePG table backed with data that resides outside of the database. You can both read from and write to the same foreign table. You can similarly use foreign tables in SQL commands as described above for external tables. Refer to [Accessing External Data with Foreign Tables](foreign/index.md) for more information about accessing external data using foreign tables.

Web-based external tables provide access to data served by an HTTP server or an operating system process. See [Creating and Using External Web Tables](external-tables/creating-and-using-web-external-tables.md) for more about web-based tables.

- **[Accessing External Data with PXF](pxf-overview.md)**
- **[PXF for WarehousePG](/pxf/6x/index.md)**
Data managed by your organization may already reside in external sources such as Hadoop, object stores, and other SQL databases. The WarehousePG Platform Extension Framework (PXF) provides access to this external data via built-in connectors that map an external data source to a WarehousePG table definition.

- **[Defining External Tables](external-tables/index.md)**
Expand Down
16 changes: 0 additions & 16 deletions docs/whpg/6x/admin_guide/external/pxf-overview.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/whpg/6x/admin_guide/load/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ WarehousePG can read from and write to several types of external data sources, i
- External web tables provide access to dynamic data. They can be backed with data from URLs accessed using the HTTP protocol or by the output of an OS script running on one or more segments.
- The `gpfdist` utility is the WarehousePG parallel file distribution program. It is an HTTP server that is used with external tables to allow WarehousePG segments to load external data in parallel, from multiple file systems. You can run multiple instances of `gpfdist` on different hosts and network interfaces and access them in parallel.
- The `gpload` utility automates the steps of a load task using `gpfdist` and a YAML-formatted control file.
- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [Accessing External Data with PXF](../external/pxf-overview.md).
- You can create readable and writable external tables with the WarehousePG Platform Extension Framework (PXF), and use these tables to load data into, or offload data from, WarehousePG. For information about using PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md).

The method you choose to load data depends on the characteristics of the source data—its location, size, format, and any transformations required.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ navigation:

Writable external tables that output data to files can use the WarehousePG parallel file server program, gpfdist, or the WarehousePG Platform Extension Framework (PXF), WarehousePG's interface to Hadoop.

Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [Accessing External Data with PXF](../../../external/pxf-overview.md) for instructions on setting up PXF for use with an external table
Use the `CREATE WRITABLE EXTERNAL TABLE` command to define the external table and specify the location and format of the output files. See [Using the WarehousePG Parallel File Server (gpfdist)](../../../external/using-gpfdist.md) for instructions on setting up gpfdist for use with an external table and [PXF for WarehousePG](/pxf/6x/index.md) for instructions on setting up PXF for use with an external table

- With a writable external table using the gpfdist protocol, the WarehousePG segments send their data to gpfdist, which writes the data to the named file. gpfdist must run on a host that the WarehousePG segments can access over the network. gpfdist points to a file location on the output host and writes data received from the WarehousePG segments to the file. To divide the output data among multiple files, list multiple gpfdist URIs in your writable external table definition.
- A writable external web table sends data to an application as a stream of data. For example, unload data from WarehousePG and send it to an application that connects to another database or ETL tool to load the data elsewhere. Writable external web tables use the `EXECUTE` clause to specify a shell command, script, or application to run on the segment hosts and accept an input stream of data. See [Defining a Command-Based Writable External Web Table](../defining-a-command-based-writable-external-web-table/index.md) for more information about using `EXECUTE` commands in a writable external table definition.
Expand Down
2 changes: 1 addition & 1 deletion docs/whpg/6x/admin_guide/managing/compression.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ You can configure support for data compression with these features and utilities

- Append-optimized tables support compressing table data. See [CREATE TABLE](../../ref_guide/sql_commands/CREATE_TABLE.md).
- User-defined data types can be defined to compress data. See [CREATE TYPE](../../ref_guide/sql_commands/CREATE_TYPE.md).
- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](../external/pxf-overview.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md).
- The external table protocols [gpfdist](../external/external-tables/gpfdist-protocol.md) ([gpfdists](../external/external-tables/gpfdists-protocol.md)), [s3](../external/external-tables/s3-protocol.md), and [pxf](/pxf/6x/index.md) support compression when accessing external data. For information about external tables, see [CREATE EXTERNAL TABLE](../../ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md).
- Workfiles (temporary spill files that are created when running a query that requires more memory than it is allocated) can be compressed. See the server configuration parameter [gp_workfile_compression](../../ref_guide/config_params/guc-list.md).
- The WarehousePG utilities gpbackup, gprestore, [gpload](../../ref_guide/utility_guide/reference/gpload.md), and [gplogfilter](../../ref_guide/utility_guide/reference/gplogfilter.md) support compression.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ description: Installation guides for tools that integrate with WarehousePG but a

These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`.

- [WarehousePG Platform Extension Framework (PXF)](../../data_sci_pkgs/install_pxf.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop.
- whpg-backup - Parallel backup and restore for WarehousePG clusters.
- whpg-backup-s3-plugin - Amazon S3 support for whpg-backup.
- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Connectors for accessing data stored in external sources such as object storage and Hadoop.
- [whpg-backup](/whpg-backup/index.md) - Parallel backup and restore for WarehousePG clusters.
- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Amazon S3 support for whpg-backup.
2 changes: 1 addition & 1 deletion docs/whpg/6x/install_guide/data_sci_pkgs/install_pxf.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@ title: Platform Extension Framework (PXF)

*Optional.* If you do not plan to use PXF, no action is necessary.

If you plan to use PXF, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for introductory PXF information.
If you plan to use PXF, refer to [PXF for WarehousePG](/pxf/6x/index.md) for installation and usage information.

**Parent topic:** [Installing Optional Extensions (WarehousePG)](index.md)
2 changes: 1 addition & 1 deletion docs/whpg/6x/install_guide/platform-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ These WarehousePG extensions are installed with WarehousePG

### Data Connectors

- WarehousePG Platform Extension Framework (PXF) - PXF provides access to Hadoop, object store, and SQL external data stores. Refer to [Accessing External Data with PXF](../admin_guide/external/pxf-overview.md) in the *WarehousePG Administrator Guide* for PXF configuration and usage information.
- WarehousePG Platform Extension Framework (PXF) - PXF provides access to Hadoop, object store, and SQL external data stores. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for PXF configuration and usage information.

- WarehousePG Connector for Apache Spark v1.6.2 - The WarehousePG Connector for Apache Spark supports high speed, parallel data transfer between WarehousePG and an Apache Spark cluster using Spark’s Scala API.

Expand Down
6 changes: 3 additions & 3 deletions docs/whpg/6x/ref_guide/modules/external_components/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,6 @@ description: Separately installed tools and frameworks that integrate with Wareh

These components are installed and managed independently from WarehousePG, and aren't activated with `CREATE EXTENSION`.

- [WarehousePG Platform Extension Framework (PXF)](../../../install_guide/data_sci_pkgs/install_pxf.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop.
- whpg-backup - Provides parallel backup and restore for WarehousePG clusters.
- whpg-backup-s3-plugin - Adds Amazon S3 support to whpg-backup.
- [WarehousePG Platform Extension Framework (PXF)](/pxf/6x/index.md) - Provides connectors for accessing data stored in external sources such as object storage and Hadoop.
- [whpg-backup](/whpg-backup/index.md) - Provides parallel backup and restore for WarehousePG clusters.
- [whpg-backup-s3-plugin](/whpg-backup/s3-plugin.md) - Adds Amazon S3 support to whpg-backup.
6 changes: 3 additions & 3 deletions docs/whpg/6x/ref_guide/sql_commands/CREATE_EXTERNAL_TABLE.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ See [Working with External Data](../../admin_guide/external/index.md) for detail

- LOCATION ('protocol://\[host\[:port]]/path/file' \[, ...])

If you use the `pxf` protocol to access an external data source, refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about the `pxf` protocol.
If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about the `pxf` protocol.

If you use the `s3` protocol to read or write to S3, refer to [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md) for additional information about the `s3` protocol `LOCATION` clause syntax.

Expand Down Expand Up @@ -240,13 +240,13 @@ See [Working with External Data](../../admin_guide/external/index.md) for detail

When the `FORMAT` clause identfies delimited text (`TEXT`) or comma separated values (`CSV`) format, formatting options are similar to those available with the PostgreSQL [COPY](COPY.md) command. If the data in the file does not use the default column delimiter, escape character, null string and so on, you must specify the additional formatting options so that the data in the external file is read correctly by WarehousePG. For information about using a custom format, see "Loading and Unloading Data" in the *WarehousePG Administrator Guide*.

If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF.
If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF.

- FORMAT 'CUSTOM' (formatter=formatter_specification)

Specifies a custom data format. The formatter_specification specifies the function to use to format the data, followed by comma-separated parameters to the formatter function. The length of the formatter specification, the string including `Formatter=`, can be up to approximately 50K bytes.

If you use the `pxf` protocol to access an external data source, refer to [Accessing External Data with PXF](../../admin_guide/external/pxf-overview.md) for information about using PXF.
If you use the `pxf` protocol to access an external data source, refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using PXF.

For general information about using a custom format, see "Loading and Unloading Data" in the *WarehousePG Administrator Guide*.

Expand Down
2 changes: 1 addition & 1 deletion docs/whpg/6x/ref_guide/sql_commands/CREATE_PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ For information about creating and enabling a custom data access protocol, see "

## Notes

WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md#configuring-the-s3-protocol) for information about enabling the `S3` protocol. Refer to [pxf:// Protocol](../../admin_guide/external/external-tables/pxf-protocol.md) for information about using the `pxf` protocol.
WarehousePG handles external tables of type `file`, `gpfdist`, and `gpfdists` internally. See [s3:// Protocol](../../admin_guide/external/external-tables/s3-protocol.md#configuring-the-s3-protocol) for information about enabling the `S3` protocol. Refer to [PXF for WarehousePG](/pxf/6x/index.md) for information about using the `pxf` protocol.

Any shared library that implements a data access protocol must be located in the same location on all WarehousePG segment hosts. For example, the shared library can be in a location specified by the operating system environment variable `LD_LIBRARY_PATH` on all hosts. You can also specify the location when you define the handler function. For example, when you define the `s3` protocol in the `CREATE PROTOCOL` command, you specify `$libdir/gps3ext.so` as the location of the shared object, where `$libdir` is located at `$GPHOME/lib`.

Expand Down
2 changes: 1 addition & 1 deletion docs/whpg/6x/security-guide/SecuringGPDB.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The essential security requirements fall into the following categories:

## Accessing a Kerberized Hadoop Cluster

You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to Configuring PXF for Secure HDFS for the procedure to perform this setup.
You can use the WarehousePG Platform Extension Framework (PXF) to read or write external tables referencing files in a Hadoop file system. If the Hadoop cluster is secured with Kerberos ("Kerberized"), you must configure WarehousePG and PXF to allow users accessing external tables to authenticate with Kerberos. Refer to [Authenticating with Kerberos](/pxf/6x/connecting/hadoop/kerberos.md) for the procedure to perform this setup.

<a id="platformhardening">

Expand Down
Loading