Skip to main content

BSHTMLLoader

This notebook provides a quick overview for getting started with BeautifulSoup4 document loader. For detailed documentation of all ModuleNameLoader features and configurations head to the API reference.

Overviewโ€‹

Integration detailsโ€‹

ClassPackageLocalSerializableJS support
BSHTMLLoaderlangchain_communityโœ…โŒโŒ

Loader featuresโ€‹

SourceDocument Lazy LoadingNative Async Support
BSHTMLLoaderโœ…โŒ

Setupโ€‹

To access BSHTMLLoader document loader you'll need to install the langchain-community integration package and the bs4 python package.

Credentialsโ€‹

No credentials are needed to use the BSHTMLLoader class.

If you want to get automated best in-class tracing of your model calls you can also set your LangSmith API key by uncommenting below:

# os.environ["LANGSMITH_API_KEY"] = getpass.getpass("Enter your LangSmith API key: ")
# os.environ["LANGSMITH_TRACING"] = "true"

Installationโ€‹

Install langchain_community and bs4.

%pip install -qU langchain_community bs4

Initializationโ€‹

Now we can instantiate our model object and load documents:

  • TODO: Update model instantiation with relevant params.
from langchain_community.document_loaders import BSHTMLLoader

loader = BSHTMLLoader(
file_path="./example_data/fake-content.html",
)
API Reference:BSHTMLLoader

Loadโ€‹

docs = loader.load()
docs[0]
Document(metadata={'source': './example_data/fake-content.html', 'title': 'Test Title'}, page_content='\nTest Title\n\n\nMy First Heading\nMy first paragraph.\n\n\n')
print(docs[0].metadata)
{'source': './example_data/fake-content.html', 'title': 'Test Title'}

Lazy Loadโ€‹

page = []
for doc in loader.lazy_load():
page.append(doc)
if len(page) >= 10:
# do some paged operation, e.g.
# index.upsert(page)

page = []
page[0]
Document(metadata={'source': './example_data/fake-content.html', 'title': 'Test Title'}, page_content='\nTest Title\n\n\nMy First Heading\nMy first paragraph.\n\n\n')

Adding separator to BS4โ€‹

We can also pass a separator to use when calling get_text on the soup

loader = BSHTMLLoader(
file_path="./example_data/fake-content.html", get_text_separator=", "
)

docs = loader.load()
print(docs[0])
page_content='
, Test Title,
,
,
, My First Heading,
, My first paragraph.,
,
,
' metadata={'source': './example_data/fake-content.html', 'title': 'Test Title'}

API referenceโ€‹

For detailed documentation of all BSHTMLLoader features and configurations head to the API reference: https://api.python.langchain.com/en/latest/document_loaders/langchain_community.document_loaders.html_bs.BSHTMLLoader.html


Was this page helpful?


You can also leave detailed feedback on GitHub.